2024-09-18T11:41:59.722 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T11:41:59.738 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T11:41:59.818 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910993 branch: wip-bharath11-testing-2024-09-18-1115-quincy description: rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} email: skanta@redhat.com first_in_suite: false job_id: '7910993' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-bharath11-testing-2024-09-18-1115-quincy ceph: conf: global: mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath11-testing-2024-09-18-1115-quincy sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 seed: 4631 sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath11-testing-2024-09-18-1115-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 targets: smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - rep_lost_unfound_delete: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_09:13:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T11:41:59.818 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa; will attempt to use it 2024-09-18T11:41:59.819 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa/tasks 2024-09-18T11:41:59.819 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T11:41:59.821 INFO:teuthology.task.internal:Checking packages... 2024-09-18T11:41:59.841 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7061ea6ca50908d104e9f3979dc615e118d9e153' 2024-09-18T11:41:59.841 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T11:41:59.841 INFO:teuthology.packaging:ref: None 2024-09-18T11:41:59.841 INFO:teuthology.packaging:tag: None 2024-09-18T11:41:59.842 INFO:teuthology.packaging:branch: wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T11:41:59.842 INFO:teuthology.packaging:sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:41:59.842 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T11:41:59.990 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:41:59.991 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T11:41:59.999 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T11:41:59.999 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T11:42:00.015 INFO:teuthology.task.internal:Saving configuration 2024-09-18T11:42:00.027 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T11:42:00.036 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T11:42:00.050 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910993', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 11:35:38.281052', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T11:42:00.050 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T11:42:00.058 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2'] 2024-09-18T11:42:00.058 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T11:42:00.096 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f004195bac0>, signals=[15]) 2024-09-18T11:42:00.096 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T11:42:00.103 INFO:teuthology.task.internal:Opening connections... 2024-09-18T11:42:00.104 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-09-18T11:42:00.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:42:00.196 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T11:42:00.202 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-09-18T11:42:00.311 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-09-18T11:42:00.311 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-09-18T11:42:00.317 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="Ubuntu" 2024-09-18T11:42:00.317 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T11:42:00.317 INFO:teuthology.orchestra.run.smithi154.stdout:ID=ubuntu 2024-09-18T11:42:00.317 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE=debian 2024-09-18T11:42:00.318 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T11:42:00.318 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="20.04" 2024-09-18T11:42:00.318 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T11:42:00.318 INFO:teuthology.orchestra.run.smithi154.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T11:42:00.318 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T11:42:00.318 INFO:teuthology.orchestra.run.smithi154.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T11:42:00.318 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_CODENAME=focal 2024-09-18T11:42:00.318 INFO:teuthology.orchestra.run.smithi154.stdout:UBUNTU_CODENAME=focal 2024-09-18T11:42:00.319 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-09-18T11:42:00.345 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T11:42:00.359 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T11:42:00.372 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T11:42:00.372 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-09-18T11:42:00.378 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T11:42:00.388 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T11:42:00.389 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-09-18T11:42:00.430 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T11:42:00.466 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T11:42:00.484 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T11:42:00.485 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T11:42:00.485 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T11:42:00.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:42:00.491 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-09-18T11:42:00.498 INFO:teuthology.orchestra.run.smithi154.stdout:5.4.0-124-generic 2024-09-18T11:42:00.498 INFO:teuthology.task.kernel:Running kernel on smithi154: 5.4.0-124-generic 2024-09-18T11:42:00.498 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get clean 2024-09-18T11:42:00.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get update 2024-09-18T11:42:00.818 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T11:42:00.828 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T11:42:00.889 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T11:42:00.948 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T11:42:01.270 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T11:42:01.351 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,598 kB] 2024-09-18T11:42:01.396 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T11:42:01.401 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T11:42:01.402 INFO:teuthology.orchestra.run.smithi154.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T11:42:01.434 INFO:teuthology.orchestra.run.smithi154.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T11:42:01.435 INFO:teuthology.orchestra.run.smithi154.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T11:42:01.439 INFO:teuthology.orchestra.run.smithi154.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T11:42:01.439 INFO:teuthology.orchestra.run.smithi154.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T11:42:01.447 INFO:teuthology.orchestra.run.smithi154.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [808 kB] 2024-09-18T11:42:01.452 INFO:teuthology.orchestra.run.smithi154.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T11:42:01.454 INFO:teuthology.orchestra.run.smithi154.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T11:42:01.455 INFO:teuthology.orchestra.run.smithi154.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T11:42:01.455 INFO:teuthology.orchestra.run.smithi154.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T11:42:01.455 INFO:teuthology.orchestra.run.smithi154.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T11:42:01.469 INFO:teuthology.orchestra.run.smithi154.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T11:42:01.470 INFO:teuthology.orchestra.run.smithi154.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T11:42:01.470 INFO:teuthology.orchestra.run.smithi154.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T11:42:01.471 INFO:teuthology.orchestra.run.smithi154.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T11:42:01.487 INFO:teuthology.orchestra.run.smithi154.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T11:42:01.487 INFO:teuthology.orchestra.run.smithi154.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T11:42:01.488 INFO:teuthology.orchestra.run.smithi154.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T11:42:01.594 INFO:teuthology.orchestra.run.smithi154.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T11:42:02.082 INFO:teuthology.orchestra.run.smithi154.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T11:42:02.096 INFO:teuthology.orchestra.run.smithi154.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T11:42:02.105 INFO:teuthology.orchestra.run.smithi154.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T11:42:02.105 INFO:teuthology.orchestra.run.smithi154.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T11:42:02.194 INFO:teuthology.orchestra.run.smithi154.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T11:42:02.195 INFO:teuthology.orchestra.run.smithi154.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T11:42:02.202 INFO:teuthology.orchestra.run.smithi154.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T11:42:02.202 INFO:teuthology.orchestra.run.smithi154.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T11:42:02.253 INFO:teuthology.orchestra.run.smithi154.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T11:42:02.264 INFO:teuthology.orchestra.run.smithi154.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T11:42:02.269 INFO:teuthology.orchestra.run.smithi154.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T11:42:02.269 INFO:teuthology.orchestra.run.smithi154.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T11:42:02.269 INFO:teuthology.orchestra.run.smithi154.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T11:42:02.341 INFO:teuthology.orchestra.run.smithi154.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T11:42:02.341 INFO:teuthology.orchestra.run.smithi154.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T11:42:05.876 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 22.0 MB in 3s (7,016 kB/s) 2024-09-18T11:42:07.504 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2024-09-18T11:42:07.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T11:42:07.608 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2024-09-18T11:42:07.826 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2024-09-18T11:42:07.827 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2024-09-18T11:42:08.026 INFO:teuthology.orchestra.run.smithi154.stdout:The following additional packages will be installed: 2024-09-18T11:42:08.026 INFO:teuthology.orchestra.run.smithi154.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T11:42:08.026 INFO:teuthology.orchestra.run.smithi154.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T11:42:08.027 INFO:teuthology.orchestra.run.smithi154.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T11:42:08.028 INFO:teuthology.orchestra.run.smithi154.stdout:Suggested packages: 2024-09-18T11:42:08.028 INFO:teuthology.orchestra.run.smithi154.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T11:42:08.028 INFO:teuthology.orchestra.run.smithi154.stdout:Recommended packages: 2024-09-18T11:42:08.028 INFO:teuthology.orchestra.run.smithi154.stdout: thermald 2024-09-18T11:42:08.091 INFO:teuthology.orchestra.run.smithi154.stdout:The following NEW packages will be installed: 2024-09-18T11:42:08.091 INFO:teuthology.orchestra.run.smithi154.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T11:42:08.092 INFO:teuthology.orchestra.run.smithi154.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T11:42:08.092 INFO:teuthology.orchestra.run.smithi154.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T11:42:08.093 INFO:teuthology.orchestra.run.smithi154.stdout:The following packages will be upgraded: 2024-09-18T11:42:08.094 INFO:teuthology.orchestra.run.smithi154.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T11:42:08.275 INFO:teuthology.orchestra.run.smithi154.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T11:42:08.276 INFO:teuthology.orchestra.run.smithi154.stdout:Need to get 77.2 MB of archives. 2024-09-18T11:42:08.276 INFO:teuthology.orchestra.run.smithi154.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T11:42:08.276 INFO:teuthology.orchestra.run.smithi154.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T11:42:09.421 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T11:42:09.694 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T11:42:10.973 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T11:42:11.049 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T11:42:11.049 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T11:42:11.328 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T11:42:11.348 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T11:42:12.063 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 77.2 MB in 3s (23.8 MB/s) 2024-09-18T11:42:12.248 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T11:42:14.011 INFO:teuthology.orchestra.run.smithi154.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T11:42:14.016 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T11:42:14.170 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:42:16.155 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T11:42:16.168 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T11:42:16.268 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:42:17.350 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T11:42:17.363 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T11:42:17.432 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:42:22.772 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T11:42:22.906 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T11:42:23.230 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T11:42:23.359 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T11:42:23.720 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T11:42:23.734 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T11:42:23.778 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T11:42:29.325 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T11:42:29.346 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T11:42:29.397 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:42:32.822 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T11:42:32.950 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T11:42:33.298 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T11:42:33.447 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:42:33.573 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T11:42:33.724 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:42:34.993 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:42:36.158 INFO:teuthology.orchestra.run.smithi154.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T11:42:36.159 INFO:teuthology.orchestra.run.smithi154.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T11:42:36.319 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:42:38.117 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T11:42:39.123 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T11:42:39.266 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T11:42:39.342 INFO:teuthology.orchestra.run.smithi154.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T11:42:39.342 INFO:teuthology.orchestra.run.smithi154.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T11:43:01.601 INFO:teuthology.orchestra.run.smithi154.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T11:43:01.602 INFO:teuthology.orchestra.run.smithi154.stdout:Sourcing file `/etc/default/grub' 2024-09-18T11:43:01.622 INFO:teuthology.orchestra.run.smithi154.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T11:43:01.651 INFO:teuthology.orchestra.run.smithi154.stdout:Generating grub configuration file ... 2024-09-18T11:43:02.031 INFO:teuthology.orchestra.run.smithi154.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T11:43:02.054 INFO:teuthology.orchestra.run.smithi154.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T11:43:02.448 INFO:teuthology.orchestra.run.smithi154.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T11:43:02.458 INFO:teuthology.orchestra.run.smithi154.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T11:43:05.229 INFO:teuthology.orchestra.run.smithi154.stdout:done 2024-09-18T11:43:05.705 DEBUG:teuthology.orchestra.run.smithi154:> dpkg -s linux-image-generic 2024-09-18T11:43:05.762 INFO:teuthology.orchestra.run.smithi154.stdout:Package: linux-image-generic 2024-09-18T11:43:05.762 INFO:teuthology.orchestra.run.smithi154.stdout:Status: install ok installed 2024-09-18T11:43:05.762 INFO:teuthology.orchestra.run.smithi154.stdout:Priority: optional 2024-09-18T11:43:05.763 INFO:teuthology.orchestra.run.smithi154.stdout:Section: kernel 2024-09-18T11:43:05.763 INFO:teuthology.orchestra.run.smithi154.stdout:Installed-Size: 20 2024-09-18T11:43:05.763 INFO:teuthology.orchestra.run.smithi154.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T11:43:05.763 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: amd64 2024-09-18T11:43:05.763 INFO:teuthology.orchestra.run.smithi154.stdout:Source: linux-meta 2024-09-18T11:43:05.763 INFO:teuthology.orchestra.run.smithi154.stdout:Version: 5.4.0.196.194 2024-09-18T11:43:05.763 INFO:teuthology.orchestra.run.smithi154.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T11:43:05.763 INFO:teuthology.orchestra.run.smithi154.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T11:43:05.763 INFO:teuthology.orchestra.run.smithi154.stdout:Recommends: thermald 2024-09-18T11:43:05.763 INFO:teuthology.orchestra.run.smithi154.stdout:Description: Generic Linux kernel image 2024-09-18T11:43:05.763 INFO:teuthology.orchestra.run.smithi154.stdout: This package will always depend on the latest generic kernel image 2024-09-18T11:43:05.763 INFO:teuthology.orchestra.run.smithi154.stdout: available. 2024-09-18T11:43:05.764 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T11:43:05.764 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T11:43:05.764 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T11:43:05.764 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T11:43:05.764 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T11:43:05.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get clean 2024-09-18T11:43:05.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get update 2024-09-18T11:43:05.973 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T11:43:05.973 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T11:43:05.981 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T11:43:05.998 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T11:43:07.806 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2024-09-18T11:43:07.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T11:43:07.905 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2024-09-18T11:43:08.124 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2024-09-18T11:43:08.125 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2024-09-18T11:43:08.364 INFO:teuthology.orchestra.run.smithi154.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T11:43:08.365 INFO:teuthology.orchestra.run.smithi154.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T11:43:08.366 DEBUG:teuthology.orchestra.run.smithi154:> dpkg -s linux-image-generic 2024-09-18T11:43:08.387 INFO:teuthology.orchestra.run.smithi154.stdout:Package: linux-image-generic 2024-09-18T11:43:08.387 INFO:teuthology.orchestra.run.smithi154.stdout:Status: install ok installed 2024-09-18T11:43:08.387 INFO:teuthology.orchestra.run.smithi154.stdout:Priority: optional 2024-09-18T11:43:08.387 INFO:teuthology.orchestra.run.smithi154.stdout:Section: kernel 2024-09-18T11:43:08.387 INFO:teuthology.orchestra.run.smithi154.stdout:Installed-Size: 20 2024-09-18T11:43:08.388 INFO:teuthology.orchestra.run.smithi154.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T11:43:08.388 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: amd64 2024-09-18T11:43:08.388 INFO:teuthology.orchestra.run.smithi154.stdout:Source: linux-meta 2024-09-18T11:43:08.388 INFO:teuthology.orchestra.run.smithi154.stdout:Version: 5.4.0.196.194 2024-09-18T11:43:08.388 INFO:teuthology.orchestra.run.smithi154.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T11:43:08.388 INFO:teuthology.orchestra.run.smithi154.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T11:43:08.388 INFO:teuthology.orchestra.run.smithi154.stdout:Recommends: thermald 2024-09-18T11:43:08.388 INFO:teuthology.orchestra.run.smithi154.stdout:Description: Generic Linux kernel image 2024-09-18T11:43:08.388 INFO:teuthology.orchestra.run.smithi154.stdout: This package will always depend on the latest generic kernel image 2024-09-18T11:43:08.388 INFO:teuthology.orchestra.run.smithi154.stdout: available. 2024-09-18T11:43:08.389 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-18T11:43:08.393 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.DPNDTBKyGa 2024-09-18T11:43:08.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub/grub.cfg /tmp/tmp.DPNDTBKyGa 2024-09-18T11:43:08.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.DPNDTBKyGa 2024-09-18T11:43:08.482 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.DPNDTBKyGa is 9KB 2024-09-18T11:43:08.496 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.DPNDTBKyGa 2024-09-18T11:43:08.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T11:43:08.558 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:43:08.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T11:43:08.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T11:43:08.589 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T11:43:08.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo update-grub 2024-09-18T11:43:10.077 INFO:teuthology.orchestra.run.smithi154.stderr:Sourcing file `/etc/default/grub' 2024-09-18T11:43:10.086 INFO:teuthology.orchestra.run.smithi154.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T11:43:10.099 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2024-09-18T11:43:10.382 INFO:teuthology.orchestra.run.smithi154.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T11:43:10.394 INFO:teuthology.orchestra.run.smithi154.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T11:43:10.799 INFO:teuthology.orchestra.run.smithi154.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T11:43:10.811 INFO:teuthology.orchestra.run.smithi154.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T11:43:12.469 INFO:teuthology.orchestra.run.smithi154.stderr:done 2024-09-18T11:43:12.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2024-09-18T11:43:42.491 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T11:43:42.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-18T11:43:42.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:44:00.833 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-09-18T11:44:09.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-18T11:44:09.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:44:12.893 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-09-18T11:44:24.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-18T11:44:24.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:44:27.965 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-09-18T11:44:42.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-18T11:44:42.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:45:14.803 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-09-18T11:45:15.402 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-18T11:45:15.402 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-196-generic"... 2024-09-18T11:45:15.402 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-09-18T11:45:15.409 INFO:teuthology.orchestra.run.smithi154.stdout:5.4.0-196-generic 2024-09-18T11:45:15.409 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T11:45:15.409 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T11:45:15.409 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T11:45:16.410 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T11:45:16.410 DEBUG:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T11:45:16.503 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2024-09-18T11:45:16.526 DEBUG:teuthology.parallel:result is None 2024-09-18T11:45:16.526 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T11:45:16.534 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T11:45:16.535 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T11:45:16.542 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T11:45:16.548 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T11:45:16.556 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T11:45:16.556 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T11:45:16.607 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T11:45:16.616 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T11:45:16.616 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T11:45:16.637 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:45:16.648 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:45:16.649 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T11:45:16.656 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T11:45:16.656 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T11:45:16.708 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T11:45:16.717 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T11:45:16.717 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T11:45:16.723 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T11:45:16.770 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T11:45:16.778 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:45:16.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T11:45:16.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-09-18T11:45:16.844 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T11:45:16.854 INFO:teuthology.task.internal:Starting timer... 2024-09-18T11:45:16.854 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T11:45:16.865 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T11:45:16.873 DEBUG:teuthology.task.selinux:Excluding smithi154: OS 'ubuntu' does not support SELinux 2024-09-18T11:45:16.874 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T11:45:16.874 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-18T11:45:16.874 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T11:45:16.874 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T11:45:16.882 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-18T11:45:16.885 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-18T11:45:17.016 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T11:45:17.027 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T11:45:17.028 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T11:50:55.689 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2024-09-18T11:50:55.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-18T11:50:55.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T11:50:55.791 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-09-18T11:50:55.918 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-18T11:50:55.919 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T11:50:55.929 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T11:50:55.930 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T11:50:55.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T11:50:55.964 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:50:55 ntpd[13674]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T11:50:55.964 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:50:55 ntpd[13674]: Command line: ntpd -gq 2024-09-18T11:50:55.965 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:50:55 ntpd[13674]: proto: precision = 0.213 usec (-22) 2024-09-18T11:50:55.965 INFO:teuthology.orchestra.run.smithi154.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T11:50:55.966 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:50:55 ntpd[13674]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T11:50:55.966 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:50:55 ntpd[13674]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T11:50:55.966 INFO:teuthology.orchestra.run.smithi154.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T11:50:55.967 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:50:55 ntpd[13674]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T11:50:55.967 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:50:55 ntpd[13674]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T11:50:55.967 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:50:55 ntpd[13674]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T11:50:55.967 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:50:55 ntpd[13674]: Listen normally on 3 enp3s0f1 172.21.15.154:123 2024-09-18T11:50:55.967 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:50:55 ntpd[13674]: Listen normally on 4 lo [::1]:123 2024-09-18T11:50:55.968 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:50:55 ntpd[13674]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7761%5]:123 2024-09-18T11:50:55.968 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:50:55 ntpd[13674]: Listening on routing socket on fd #22 for interface updates 2024-09-18T11:50:56.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:56 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:50:56.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:56 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:50:56.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:56 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:50:56.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:56 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:50:56.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:56 ntpd[13674]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:50:56.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:56 ntpd[13674]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:50:57.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:57 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:50:57.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:57 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:50:57.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:57 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:50:57.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:57 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:50:57.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:57 ntpd[13674]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:50:57.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:57 ntpd[13674]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:50:58.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:58 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:50:58.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:58 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:50:58.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:58 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:50:58.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:58 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:50:58.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:58 ntpd[13674]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:50:58.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:58 ntpd[13674]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:50:59.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:59 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:50:59.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:59 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:50:59.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:59 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:50:59.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:59 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:50:59.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:59 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:50:59.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:59 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:50:59.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:59 ntpd[13674]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:50:59.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:59 ntpd[13674]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:50:59.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:59 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:50:59.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:59 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:50:59.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:59 ntpd[13674]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:50:59.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:50:59 ntpd[13674]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:51:00.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:00 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:51:00.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:00 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:51:00.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:00 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:51:00.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:00 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:51:00.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:00 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:51:00.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:00 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:51:00.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:00 ntpd[13674]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:51:00.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:00 ntpd[13674]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:51:01.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:01 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:51:01.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:01 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:51:01.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:01 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:51:01.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:01 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:51:01.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:01 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:51:01.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:01 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:51:01.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:01 ntpd[13674]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:51:01.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:01 ntpd[13674]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:51:01.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:01 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:51:01.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:01 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:51:01.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:01 ntpd[13674]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:51:01.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:01 ntpd[13674]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:51:02.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:02 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:51:02.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:02 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:51:02.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:02 ntpd[13674]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T11:51:02.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:02 ntpd[13674]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T11:51:02.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:02 ntpd[13674]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T11:51:02.965 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:02 ntpd[13674]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T11:51:02.966 INFO:teuthology.orchestra.run.smithi154.stdout:18 Sep 11:51:02 ntpd[13674]: ntpd: time slew -0.002030 s 2024-09-18T11:51:02.966 INFO:teuthology.orchestra.run.smithi154.stdout:ntpd: time slew -0.002030s 2024-09-18T11:51:02.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:02 ntpd[13674]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T11:51:02.966 INFO:teuthology.orchestra.run.smithi154.stderr:18 Sep 11:51:02 ntpd[13674]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T11:51:03.033 INFO:teuthology.orchestra.run.smithi154.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T11:51:03.033 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================== 2024-09-18T11:51:03.033 INFO:teuthology.orchestra.run.smithi154.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T11:51:03.033 INFO:teuthology.orchestra.run.smithi154.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T11:51:03.033 INFO:teuthology.orchestra.run.smithi154.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T11:51:03.033 INFO:teuthology.orchestra.run.smithi154.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T11:51:03.034 INFO:teuthology.run_tasks:Running task install... 2024-09-18T11:51:03.041 DEBUG:teuthology.task.install:project ceph 2024-09-18T11:51:03.042 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}} 2024-09-18T11:51:03.042 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'} 2024-09-18T11:51:03.042 INFO:teuthology.task.install:Using flavor: default 2024-09-18T11:51:03.047 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T11:51:03.047 INFO:teuthology.task.install:extra packages: [] 2024-09-18T11:51:03.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-key list | grep Ceph 2024-09-18T11:51:03.147 INFO:teuthology.orchestra.run.smithi154.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T11:51:03.207 INFO:teuthology.orchestra.run.smithi154.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T11:51:03.207 INFO:teuthology.orchestra.run.smithi154.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T11:51:03.208 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-18T11:51:03.209 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:51:03.359 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default/ 2024-09-18T11:51:03.359 INFO:teuthology.task.install.deb:Package version is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:51:03.473 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:51:03.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T11:51:03.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get update 2024-09-18T11:51:03.676 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T11:51:03.730 INFO:teuthology.orchestra.run.smithi154.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal InRelease 2024-09-18T11:51:03.752 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T11:51:03.776 INFO:teuthology.orchestra.run.smithi154.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T11:51:03.794 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T11:51:03.797 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main i386 Packages [3,390 B] 2024-09-18T11:51:03.838 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-18T11:51:03.876 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T11:51:03.961 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T11:51:04.346 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 28.4 kB in 1s (44.2 kB/s) 2024-09-18T11:51:05.608 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2024-09-18T11:51:05.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1645-g7061ea6c-1focal cephadm=17.2.7-1645-g7061ea6c-1focal ceph-mds=17.2.7-1645-g7061ea6c-1focal ceph-mgr=17.2.7-1645-g7061ea6c-1focal ceph-common=17.2.7-1645-g7061ea6c-1focal ceph-fuse=17.2.7-1645-g7061ea6c-1focal ceph-test=17.2.7-1645-g7061ea6c-1focal ceph-volume=17.2.7-1645-g7061ea6c-1focal radosgw=17.2.7-1645-g7061ea6c-1focal python3-rados=17.2.7-1645-g7061ea6c-1focal python3-rgw=17.2.7-1645-g7061ea6c-1focal python3-cephfs=17.2.7-1645-g7061ea6c-1focal python3-rbd=17.2.7-1645-g7061ea6c-1focal libcephfs2=17.2.7-1645-g7061ea6c-1focal libcephfs-dev=17.2.7-1645-g7061ea6c-1focal librados2=17.2.7-1645-g7061ea6c-1focal librbd1=17.2.7-1645-g7061ea6c-1focal rbd-fuse=17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:51:05.707 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2024-09-18T11:51:05.914 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2024-09-18T11:51:05.915 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2024-09-18T11:51:06.109 INFO:teuthology.orchestra.run.smithi154.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T11:51:06.110 INFO:teuthology.orchestra.run.smithi154.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T11:51:06.110 INFO:teuthology.orchestra.run.smithi154.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T11:51:06.111 INFO:teuthology.orchestra.run.smithi154.stdout:The following additional packages will be installed: 2024-09-18T11:51:06.111 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T11:51:06.111 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T11:51:06.111 INFO:teuthology.orchestra.run.smithi154.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T11:51:06.111 INFO:teuthology.orchestra.run.smithi154.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T11:51:06.111 INFO:teuthology.orchestra.run.smithi154.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T11:51:06.112 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T11:51:06.112 INFO:teuthology.orchestra.run.smithi154.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-18T11:51:06.112 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-18T11:51:06.112 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-18T11:51:06.112 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-18T11:51:06.112 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-18T11:51:06.112 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-18T11:51:06.113 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-18T11:51:06.113 INFO:teuthology.orchestra.run.smithi154.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-18T11:51:06.113 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-18T11:51:06.113 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-18T11:51:06.113 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-18T11:51:06.113 INFO:teuthology.orchestra.run.smithi154.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-18T11:51:06.113 INFO:teuthology.orchestra.run.smithi154.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-18T11:51:06.113 INFO:teuthology.orchestra.run.smithi154.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-18T11:51:06.113 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-18T11:51:06.113 INFO:teuthology.orchestra.run.smithi154.stdout: socat xmlstarlet 2024-09-18T11:51:06.114 INFO:teuthology.orchestra.run.smithi154.stdout:Suggested packages: 2024-09-18T11:51:06.114 INFO:teuthology.orchestra.run.smithi154.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T11:51:06.114 INFO:teuthology.orchestra.run.smithi154.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T11:51:06.114 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T11:51:06.114 INFO:teuthology.orchestra.run.smithi154.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T11:51:06.114 INFO:teuthology.orchestra.run.smithi154.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T11:51:06.114 INFO:teuthology.orchestra.run.smithi154.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T11:51:06.114 INFO:teuthology.orchestra.run.smithi154.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T11:51:06.114 INFO:teuthology.orchestra.run.smithi154.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T11:51:06.114 INFO:teuthology.orchestra.run.smithi154.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T11:51:06.114 INFO:teuthology.orchestra.run.smithi154.stdout:Recommended packages: 2024-09-18T11:51:06.115 INFO:teuthology.orchestra.run.smithi154.stdout: btrfs-tools 2024-09-18T11:51:06.234 INFO:teuthology.orchestra.run.smithi154.stdout:The following NEW packages will be installed: 2024-09-18T11:51:06.234 INFO:teuthology.orchestra.run.smithi154.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T11:51:06.234 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T11:51:06.234 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T11:51:06.235 INFO:teuthology.orchestra.run.smithi154.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-18T11:51:06.236 INFO:teuthology.orchestra.run.smithi154.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-18T11:51:06.237 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-18T11:51:06.237 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-18T11:51:06.237 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T11:51:06.237 INFO:teuthology.orchestra.run.smithi154.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T11:51:06.237 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T11:51:06.237 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T11:51:06.237 INFO:teuthology.orchestra.run.smithi154.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-18T11:51:06.237 INFO:teuthology.orchestra.run.smithi154.stdout:The following packages will be upgraded: 2024-09-18T11:51:06.238 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 librbd1 2024-09-18T11:51:06.387 INFO:teuthology.orchestra.run.smithi154.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T11:51:06.387 INFO:teuthology.orchestra.run.smithi154.stdout:Need to get 201 MB of archives. 2024-09-18T11:51:06.387 INFO:teuthology.orchestra.run.smithi154.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-18T11:51:06.388 INFO:teuthology.orchestra.run.smithi154.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1645-g7061ea6c-1focal [3,507 kB] 2024-09-18T11:51:06.430 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T11:51:06.584 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1645-g7061ea6c-1focal [3,562 kB] 2024-09-18T11:51:06.657 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1645-g7061ea6c-1focal [761 kB] 2024-09-18T11:51:06.666 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1645-g7061ea6c-1focal [414 kB] 2024-09-18T11:51:06.670 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1645-g7061ea6c-1focal [22.6 kB] 2024-09-18T11:51:06.671 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1645-g7061ea6c-1focal [200 kB] 2024-09-18T11:51:06.682 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1645-g7061ea6c-1focal [49.1 kB] 2024-09-18T11:51:06.683 INFO:teuthology.orchestra.run.smithi154.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1645-g7061ea6c-1focal [390 kB] 2024-09-18T11:51:06.687 INFO:teuthology.orchestra.run.smithi154.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1645-g7061ea6c-1focal [5,637 kB] 2024-09-18T11:51:06.776 INFO:teuthology.orchestra.run.smithi154.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1645-g7061ea6c-1focal [116 kB] 2024-09-18T11:51:06.777 INFO:teuthology.orchestra.run.smithi154.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1645-g7061ea6c-1focal [479 kB] 2024-09-18T11:51:06.783 INFO:teuthology.orchestra.run.smithi154.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1645-g7061ea6c-1focal [27.0 MB] 2024-09-18T11:51:06.841 INFO:teuthology.orchestra.run.smithi154.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T11:51:06.848 INFO:teuthology.orchestra.run.smithi154.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T11:51:06.893 INFO:teuthology.orchestra.run.smithi154.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T11:51:06.907 INFO:teuthology.orchestra.run.smithi154.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T11:51:07.038 INFO:teuthology.orchestra.run.smithi154.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T11:51:07.044 INFO:teuthology.orchestra.run.smithi154.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T11:51:07.074 INFO:teuthology.orchestra.run.smithi154.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T11:51:07.222 INFO:teuthology.orchestra.run.smithi154.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T11:51:07.300 INFO:teuthology.orchestra.run.smithi154.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T11:51:07.365 INFO:teuthology.orchestra.run.smithi154.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T11:51:07.365 INFO:teuthology.orchestra.run.smithi154.stdout:Get:24 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1645-g7061ea6c-1focal [5,177 kB] 2024-09-18T11:51:07.369 INFO:teuthology.orchestra.run.smithi154.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T11:51:07.370 INFO:teuthology.orchestra.run.smithi154.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T11:51:07.376 INFO:teuthology.orchestra.run.smithi154.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T11:51:07.377 INFO:teuthology.orchestra.run.smithi154.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T11:51:07.377 INFO:teuthology.orchestra.run.smithi154.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T11:51:07.378 INFO:teuthology.orchestra.run.smithi154.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T11:51:07.379 INFO:teuthology.orchestra.run.smithi154.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T11:51:07.391 INFO:teuthology.orchestra.run.smithi154.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T11:51:07.413 INFO:teuthology.orchestra.run.smithi154.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1645-g7061ea6c-1focal [223 kB] 2024-09-18T11:51:07.414 INFO:teuthology.orchestra.run.smithi154.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1645-g7061ea6c-1focal [157 kB] 2024-09-18T11:51:07.416 INFO:teuthology.orchestra.run.smithi154.stdout:Get:35 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1645-g7061ea6c-1focal [1,594 kB] 2024-09-18T11:51:07.438 INFO:teuthology.orchestra.run.smithi154.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1645-g7061ea6c-1focal [4,688 kB] 2024-09-18T11:51:07.464 INFO:teuthology.orchestra.run.smithi154.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T11:51:07.465 INFO:teuthology.orchestra.run.smithi154.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T11:51:07.466 INFO:teuthology.orchestra.run.smithi154.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T11:51:07.468 INFO:teuthology.orchestra.run.smithi154.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T11:51:07.468 INFO:teuthology.orchestra.run.smithi154.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T11:51:07.469 INFO:teuthology.orchestra.run.smithi154.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T11:51:07.471 INFO:teuthology.orchestra.run.smithi154.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T11:51:07.472 INFO:teuthology.orchestra.run.smithi154.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T11:51:07.476 INFO:teuthology.orchestra.run.smithi154.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T11:51:07.515 INFO:teuthology.orchestra.run.smithi154.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1645-g7061ea6c-1focal [23.1 MB] 2024-09-18T11:51:07.559 INFO:teuthology.orchestra.run.smithi154.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T11:51:07.560 INFO:teuthology.orchestra.run.smithi154.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T11:51:07.569 INFO:teuthology.orchestra.run.smithi154.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T11:51:07.570 INFO:teuthology.orchestra.run.smithi154.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T11:51:07.571 INFO:teuthology.orchestra.run.smithi154.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T11:51:07.572 INFO:teuthology.orchestra.run.smithi154.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T11:51:07.573 INFO:teuthology.orchestra.run.smithi154.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T11:51:07.575 INFO:teuthology.orchestra.run.smithi154.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T11:51:07.645 INFO:teuthology.orchestra.run.smithi154.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T11:51:07.645 INFO:teuthology.orchestra.run.smithi154.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T11:51:07.728 INFO:teuthology.orchestra.run.smithi154.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T11:51:07.730 INFO:teuthology.orchestra.run.smithi154.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T11:51:07.732 INFO:teuthology.orchestra.run.smithi154.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T11:51:07.733 INFO:teuthology.orchestra.run.smithi154.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T11:51:07.734 INFO:teuthology.orchestra.run.smithi154.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T11:51:08.030 INFO:teuthology.orchestra.run.smithi154.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1645-g7061ea6c-1focal [3,936 B] 2024-09-18T11:51:08.031 INFO:teuthology.orchestra.run.smithi154.stdout:Get:63 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1645-g7061ea6c-1focal [872 kB] 2024-09-18T11:51:08.046 INFO:teuthology.orchestra.run.smithi154.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1645-g7061ea6c-1focal [2,416 kB] 2024-09-18T11:51:08.063 INFO:teuthology.orchestra.run.smithi154.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1645-g7061ea6c-1focal [81.4 kB] 2024-09-18T11:51:08.064 INFO:teuthology.orchestra.run.smithi154.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1645-g7061ea6c-1focal [120 kB] 2024-09-18T11:51:08.065 INFO:teuthology.orchestra.run.smithi154.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1645-g7061ea6c-1focal [6,653 kB] 2024-09-18T11:51:08.068 INFO:teuthology.orchestra.run.smithi154.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T11:51:08.106 INFO:teuthology.orchestra.run.smithi154.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T11:51:08.154 INFO:teuthology.orchestra.run.smithi154.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T11:51:08.157 INFO:teuthology.orchestra.run.smithi154.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1645-g7061ea6c-1focal [7,884 kB] 2024-09-18T11:51:08.179 INFO:teuthology.orchestra.run.smithi154.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T11:51:08.180 INFO:teuthology.orchestra.run.smithi154.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T11:51:08.180 INFO:teuthology.orchestra.run.smithi154.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T11:51:08.181 INFO:teuthology.orchestra.run.smithi154.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T11:51:08.181 INFO:teuthology.orchestra.run.smithi154.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T11:51:08.182 INFO:teuthology.orchestra.run.smithi154.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T11:51:08.183 INFO:teuthology.orchestra.run.smithi154.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T11:51:08.244 INFO:teuthology.orchestra.run.smithi154.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T11:51:08.244 INFO:teuthology.orchestra.run.smithi154.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T11:51:08.286 INFO:teuthology.orchestra.run.smithi154.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1645-g7061ea6c-1focal [4,056 B] 2024-09-18T11:51:08.287 INFO:teuthology.orchestra.run.smithi154.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1645-g7061ea6c-1focal [53.4 MB] 2024-09-18T11:51:08.329 INFO:teuthology.orchestra.run.smithi154.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T11:51:08.334 INFO:teuthology.orchestra.run.smithi154.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T11:51:08.337 INFO:teuthology.orchestra.run.smithi154.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T11:51:08.340 INFO:teuthology.orchestra.run.smithi154.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T11:51:08.341 INFO:teuthology.orchestra.run.smithi154.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T11:51:08.345 INFO:teuthology.orchestra.run.smithi154.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T11:51:08.348 INFO:teuthology.orchestra.run.smithi154.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T11:51:08.425 INFO:teuthology.orchestra.run.smithi154.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T11:51:08.430 INFO:teuthology.orchestra.run.smithi154.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T11:51:08.603 INFO:teuthology.orchestra.run.smithi154.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T11:51:08.604 INFO:teuthology.orchestra.run.smithi154.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T11:51:08.605 INFO:teuthology.orchestra.run.smithi154.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T11:51:08.606 INFO:teuthology.orchestra.run.smithi154.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T11:51:08.606 INFO:teuthology.orchestra.run.smithi154.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T11:51:08.607 INFO:teuthology.orchestra.run.smithi154.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T11:51:08.622 INFO:teuthology.orchestra.run.smithi154.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T11:51:08.623 INFO:teuthology.orchestra.run.smithi154.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T11:51:08.652 INFO:teuthology.orchestra.run.smithi154.stdout:Get:100 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T11:51:08.689 INFO:teuthology.orchestra.run.smithi154.stdout:Get:101 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T11:51:08.772 INFO:teuthology.orchestra.run.smithi154.stdout:Get:102 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T11:51:08.773 INFO:teuthology.orchestra.run.smithi154.stdout:Get:103 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T11:51:08.774 INFO:teuthology.orchestra.run.smithi154.stdout:Get:104 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T11:51:08.777 INFO:teuthology.orchestra.run.smithi154.stdout:Get:105 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T11:51:08.779 INFO:teuthology.orchestra.run.smithi154.stdout:Get:106 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T11:51:09.399 INFO:teuthology.orchestra.run.smithi154.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1645-g7061ea6c-1focal [231 kB] 2024-09-18T11:51:09.404 INFO:teuthology.orchestra.run.smithi154.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1645-g7061ea6c-1focal [21.9 kB] 2024-09-18T11:51:09.404 INFO:teuthology.orchestra.run.smithi154.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1645-g7061ea6c-1focal [17.2 MB] 2024-09-18T11:51:09.701 INFO:teuthology.orchestra.run.smithi154.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1645-g7061ea6c-1focal [80.6 kB] 2024-09-18T11:51:10.343 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 201 MB in 3s (58.2 MB/s) 2024-09-18T11:51:10.469 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T11:51:10.525 INFO:teuthology.orchestra.run.smithi154.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T11:51:10.530 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T11:51:10.618 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T11:51:10.986 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../001-librbd1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:11.105 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librbd1 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T11:51:11.835 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../002-librados2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:11.943 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librados2 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T11:51:12.613 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T11:51:12.630 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:12.671 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:13.032 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rados. 2024-09-18T11:51:13.051 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:13.090 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:13.426 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T11:51:13.446 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:51:13.509 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:13.762 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T11:51:13.779 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:13.837 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:14.198 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T11:51:14.214 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:51:14.256 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:14.609 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T11:51:14.625 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T11:51:14.667 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T11:51:14.928 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T11:51:14.947 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:14.986 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:15.356 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T11:51:15.372 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T11:51:15.414 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T11:51:15.800 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T11:51:15.819 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T11:51:15.858 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T11:51:16.220 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T11:51:16.237 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T11:51:16.278 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T11:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T11:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T11:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T11:51:17.100 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T11:51:17.119 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T11:51:17.158 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T11:51:17.654 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T11:51:17.670 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T11:51:17.747 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:51:18.290 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T11:51:18.306 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T11:51:18.382 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:51:18.768 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T11:51:18.784 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T11:51:18.826 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:51:19.229 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T11:51:19.247 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T11:51:19.312 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T11:51:19.699 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librgw2. 2024-09-18T11:51:19.716 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../019-librgw2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:19.757 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:20.568 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T11:51:20.587 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:20.667 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:21.029 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package liboath0. 2024-09-18T11:51:21.046 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T11:51:21.087 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T11:51:21.415 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T11:51:21.432 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:21.481 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:22.018 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-common. 2024-09-18T11:51:22.033 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:22.076 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:24.848 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-base. 2024-09-18T11:51:24.864 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:24.917 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:25.861 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T11:51:25.877 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T11:51:25.919 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T11:51:26.314 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T11:51:26.334 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T11:51:26.397 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T11:51:26.683 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T11:51:26.700 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-18T11:51:26.758 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T11:51:26.986 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T11:51:27.003 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:51:27.069 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:27.466 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T11:51:27.483 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T11:51:27.524 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T11:51:27.835 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T11:51:27.854 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T11:51:27.893 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T11:51:28.246 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T11:51:28.266 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T11:51:28.304 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T11:51:28.649 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T11:51:28.667 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T11:51:28.732 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T11:51:29.077 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T11:51:29.095 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T11:51:29.135 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T11:51:29.455 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T11:51:29.472 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T11:51:29.513 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T11:51:29.849 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-mako. 2024-09-18T11:51:29.869 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T11:51:29.907 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T11:51:30.261 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T11:51:30.276 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T11:51:30.319 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T11:51:30.660 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T11:51:30.676 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T11:51:30.751 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T11:51:31.072 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-webob. 2024-09-18T11:51:31.088 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T11:51:31.130 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T11:51:31.475 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T11:51:31.492 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T11:51:31.533 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T11:51:31.861 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T11:51:31.877 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T11:51:31.919 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T11:51:32.347 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T11:51:32.366 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T11:51:32.408 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T11:51:32.759 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T11:51:32.776 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T11:51:32.816 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T11:51:33.153 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-paste. 2024-09-18T11:51:33.171 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T11:51:33.211 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T11:51:33.556 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T11:51:33.573 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T11:51:33.631 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T11:51:33.969 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T11:51:33.985 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T11:51:34.027 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T11:51:34.349 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T11:51:34.367 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T11:51:34.407 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T11:51:34.785 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T11:51:34.803 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T11:51:34.843 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T11:51:35.255 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T11:51:35.274 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T11:51:35.313 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T11:51:35.700 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T11:51:35.716 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:35.758 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:36.128 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T11:51:36.147 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:36.211 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:36.731 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T11:51:36.748 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:36.792 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:37.593 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T11:51:37.611 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:37.651 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:39.729 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph. 2024-09-18T11:51:39.747 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../053-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:39.811 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:41.032 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T11:51:41.048 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:41.107 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:41.485 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T11:51:41.505 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:41.543 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:42.105 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package cephadm. 2024-09-18T11:51:42.121 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../056-cephadm_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:42.162 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:42.474 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T11:51:42.490 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T11:51:42.524 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T11:51:42.884 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T11:51:42.902 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:51:42.943 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:43.280 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T11:51:43.298 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T11:51:43.329 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T11:51:43.584 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T11:51:43.600 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T11:51:43.642 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T11:51:43.945 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T11:51:43.963 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T11:51:44.003 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T11:51:44.307 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-routes. 2024-09-18T11:51:44.323 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-18T11:51:44.356 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T11:51:44.710 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T11:51:44.726 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:51:44.768 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:46.098 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T11:51:46.118 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T11:51:46.181 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T11:51:46.517 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T11:51:46.538 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T11:51:46.575 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T11:51:46.920 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T11:51:46.936 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T11:51:46.978 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T11:51:48.312 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T11:51:48.331 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T11:51:48.370 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T11:51:48.982 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T11:51:49.002 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T11:51:49.031 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T11:51:49.402 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T11:51:49.419 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T11:51:49.451 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T11:51:50.276 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T11:51:50.295 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:51:50.334 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:51.454 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T11:51:51.470 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T11:51:51.512 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T11:51:51.932 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T11:51:51.949 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T11:51:51.990 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T11:51:52.335 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T11:51:52.355 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T11:51:52.394 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T11:51:52.738 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T11:51:52.754 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-18T11:51:52.796 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T11:51:53.132 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T11:51:53.150 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T11:51:53.190 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T11:51:53.536 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T11:51:53.553 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T11:51:53.594 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T11:51:53.922 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T11:51:53.938 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T11:51:53.996 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T11:51:54.358 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T11:51:54.376 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T11:51:54.416 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T11:51:54.773 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T11:51:54.791 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T11:51:54.831 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T11:51:55.184 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T11:51:55.203 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T11:51:55.261 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T11:51:56.458 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T11:51:56.476 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:51:56.516 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:51:56.878 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T11:51:56.896 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T11:51:56.936 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T11:51:57.323 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T11:51:57.339 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T11:51:57.406 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T11:51:57.679 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package jq. 2024-09-18T11:51:57.696 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T11:51:57.770 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T11:51:58.023 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package socat. 2024-09-18T11:51:58.040 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T11:51:58.107 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T11:51:58.463 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T11:51:58.480 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T11:51:58.527 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T11:51:58.884 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-test. 2024-09-18T11:51:58.903 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:51:58.942 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:03.504 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T11:52:03.522 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T11:52:03.562 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:03.900 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T11:52:03.919 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T11:52:03.983 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T11:52:04.520 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T11:52:04.539 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:52:04.578 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:05.006 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T11:52:05.026 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T11:52:05.082 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T11:52:05.436 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T11:52:05.453 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T11:52:05.494 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T11:52:07.246 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T11:52:07.266 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T11:52:07.295 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T11:52:07.665 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T11:52:07.682 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T11:52:07.748 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T11:52:08.067 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-attr. 2024-09-18T11:52:08.088 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-18T11:52:08.125 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T11:52:08.462 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T11:52:08.479 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T11:52:08.520 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T11:52:08.856 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T11:52:08.873 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T11:52:08.914 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T11:52:09.284 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T11:52:09.301 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T11:52:09.367 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T11:52:09.862 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T11:52:09.878 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T11:52:09.920 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T11:52:10.240 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T11:52:10.259 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T11:52:10.290 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T11:52:10.635 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T11:52:10.655 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T11:52:10.692 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T11:52:11.079 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-py. 2024-09-18T11:52:11.097 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T11:52:11.137 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T11:52:11.473 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T11:52:11.492 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T11:52:11.531 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T11:52:11.912 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T11:52:11.929 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T11:52:11.970 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T11:52:12.290 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T11:52:12.307 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T11:52:12.365 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T11:52:12.726 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T11:52:12.744 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T11:52:12.784 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T11:52:13.064 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T11:52:13.081 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T11:52:13.122 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T11:52:13.885 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package radosgw. 2024-09-18T11:52:13.901 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../108-radosgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:52:13.943 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:15.691 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T11:52:15.708 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T11:52:15.749 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:16.076 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T11:52:16.370 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T11:52:16.496 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T11:52:16.648 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T11:52:16.849 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:17.504 INFO:teuthology.orchestra.run.smithi154.stdout:Adding system user cephadm....done 2024-09-18T11:52:17.682 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T11:52:17.799 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T11:52:18.210 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T11:52:18.527 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T11:52:18.653 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T11:52:18.779 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T11:52:19.140 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T11:52:19.583 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T11:52:19.876 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T11:52:20.211 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T11:52:20.528 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T11:52:20.855 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:21.181 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T11:52:21.307 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T11:52:21.591 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T11:52:21.875 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T11:52:22.001 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T11:52:22.386 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T11:52:22.712 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T11:52:23.047 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T11:52:23.406 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T11:52:23.716 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T11:52:24.058 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T11:52:24.368 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T11:52:24.652 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T11:52:24.802 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T11:52:24.929 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T11:52:25.214 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T11:52:25.607 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T11:52:25.975 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T11:52:26.369 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T11:52:26.520 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:26.838 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:52:27.072 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T11:52:27.840 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T11:52:27.966 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T11:52:28.117 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T11:52:28.393 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T11:52:28.736 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:52:28.896 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T11:52:29.022 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T11:52:29.382 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T11:52:29.683 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T11:52:29.959 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T11:52:30.085 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T11:52:30.377 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T11:52:30.712 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T11:52:31.047 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T11:52:31.442 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T11:52:31.733 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T11:52:31.883 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T11:52:32.193 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T11:52:32.553 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T11:52:32.679 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T11:52:32.838 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T11:52:33.156 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T11:52:33.441 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librados2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:33.566 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T11:52:35.047 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:35.173 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T11:52:35.299 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T11:52:35.692 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T11:52:36.010 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T11:52:36.336 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T11:52:36.471 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T11:52:36.805 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:36.965 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T11:52:37.308 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:37.451 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T11:52:37.777 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T11:52:38.824 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librbd1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:38.983 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T11:52:39.109 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T11:52:39.235 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T11:52:39.395 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T11:52:39.705 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T11:52:39.831 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T11:52:40.132 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:40.291 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:40.519 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T11:52:40.520 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T11:52:41.009 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:41.220 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T11:52:41.555 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T11:52:41.890 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T11:52:42.293 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:42.419 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:42.569 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:42.704 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:42.830 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T11:52:43.157 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T11:52:43.492 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:43.643 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:43.769 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T11:52:44.689 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:45.300 INFO:teuthology.orchestra.run.smithi154.stdout:Adding group ceph....done 2024-09-18T11:52:45.702 INFO:teuthology.orchestra.run.smithi154.stdout:Adding system user ceph....done 2024-09-18T11:52:45.878 INFO:teuthology.orchestra.run.smithi154.stdout:Setting system user ceph properties....done 2024-09-18T11:52:45.903 INFO:teuthology.orchestra.run.smithi154.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T11:52:46.056 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T11:52:46.374 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T11:52:46.824 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:46.975 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:47.249 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T11:52:47.249 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T11:52:47.919 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:48.304 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T11:52:48.763 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:49.009 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T11:52:49.009 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T11:52:49.456 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:49.712 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T11:52:49.712 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T11:52:50.140 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:50.679 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T11:52:50.679 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T11:52:51.142 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:51.302 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:51.444 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:51.697 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T11:52:51.698 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T11:52:52.179 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:52.297 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:52.406 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:52.582 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T11:52:52.709 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T11:52:52.894 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T11:52:53.196 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T11:52:55.999 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T11:52:56.685 INFO:teuthology.orchestra.run.smithi154.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T11:52:56.691 DEBUG:teuthology.parallel:result is None 2024-09-18T11:52:56.692 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T11:52:56.841 DEBUG:teuthology.orchestra.run.smithi154:> dpkg-query -W -f '${Version}' ceph 2024-09-18T11:52:56.865 INFO:teuthology.orchestra.run.smithi154.stdout:17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:52:56.866 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T11:52:56.866 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645-g7061ea6c-1focal is installed. 2024-09-18T11:52:56.868 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T11:52:56.868 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:52:56.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T11:52:56.923 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T11:52:56.924 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:52:56.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T11:52:56.933 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T11:52:56.949 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T11:52:56.949 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:52:56.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T11:52:57.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T11:52:57.016 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T11:52:57.017 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:52:57.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T11:52:57.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T11:52:57.083 INFO:teuthology.run_tasks:Running task ceph... 2024-09-18T11:52:57.165 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-18T11:52:57.165 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /var/log/ceph 2024-09-18T11:52:57.180 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-18T11:52:57.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T11:52:57.239 INFO:tasks.ceph:Creating extra log directories... 2024-09-18T11:52:57.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T11:52:57.259 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-18T11:52:57.260 INFO:tasks.ceph:config {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-18T11:52:57.260 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910993', 'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'description': 'rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7910993', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2']], 'seed': 4631, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'targets': {'smithi154.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'cluster': 'ceph'}}, {'rep_lost_unfound_delete': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-18_09:13:00', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-18T11:52:57.260 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T11:52:57.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T11:52:57.323 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:52:57.324 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T11:52:57.328 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T11:52:57.328 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2024-09-18T11:52:57.333 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T11:52:57.333 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:52:57.333 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 358 Links: 1 2024-09-18T11:52:57.333 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:52:57.333 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-18 11:49:28.917598619 +0000 2024-09-18T11:52:57.333 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-18 11:49:28.717603233 +0000 2024-09-18T11:52:57.333 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-18 11:49:28.717603233 +0000 2024-09-18T11:52:57.333 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2024-09-18T11:52:57.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T11:52:57.388 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-18T11:52:57.388 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-18T11:52:57.388 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000455907 s, 1.1 MB/s 2024-09-18T11:52:57.389 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T11:52:57.398 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2024-09-18T11:52:57.405 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T11:52:57.405 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:52:57.405 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 364 Links: 1 2024-09-18T11:52:57.405 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:52:57.405 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-18 11:49:29.133593636 +0000 2024-09-18T11:52:57.405 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-18 11:49:29.129593729 +0000 2024-09-18T11:52:57.405 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-18 11:49:29.129593729 +0000 2024-09-18T11:52:57.405 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2024-09-18T11:52:57.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T11:52:57.460 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-18T11:52:57.460 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-18T11:52:57.461 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000367032 s, 1.4 MB/s 2024-09-18T11:52:57.461 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T11:52:57.470 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2024-09-18T11:52:57.477 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T11:52:57.477 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:52:57.477 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 372 Links: 1 2024-09-18T11:52:57.477 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:52:57.477 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-18 11:49:29.773578873 +0000 2024-09-18T11:52:57.477 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-18 11:49:29.561583763 +0000 2024-09-18T11:52:57.477 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-18 11:49:29.561583763 +0000 2024-09-18T11:52:57.477 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2024-09-18T11:52:57.478 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T11:52:57.532 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-18T11:52:57.532 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-18T11:52:57.532 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000426028 s, 1.2 MB/s 2024-09-18T11:52:57.533 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T11:52:57.542 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2024-09-18T11:52:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T11:52:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T11:52:57.549 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 377 Links: 1 2024-09-18T11:52:57.549 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T11:52:57.549 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-18 11:49:30.201569001 +0000 2024-09-18T11:52:57.549 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-18 11:49:29.985573983 +0000 2024-09-18T11:52:57.549 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-18 11:49:29.985573983 +0000 2024-09-18T11:52:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2024-09-18T11:52:57.550 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T11:52:57.604 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-18T11:52:57.604 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-18T11:52:57.604 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000367036 s, 1.4 MB/s 2024-09-18T11:52:57.605 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T11:52:57.614 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T11:52:57.614 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-18T11:52:57.614 INFO:tasks.ceph:Generating config... 2024-09-18T11:52:57.616 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-18T11:52:57.616 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-18T11:52:57.616 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-18T11:52:57.616 INFO:tasks.ceph:[global] ms type = async 2024-09-18T11:52:57.616 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-18T11:52:57.616 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-18T11:52:57.616 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-18T11:52:57.616 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-18T11:52:57.616 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-18T11:52:57.617 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-18T11:52:57.617 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-18T11:52:57.618 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-18T11:52:57.618 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-18T11:52:57.618 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-18T11:52:57.618 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-18T11:52:57.618 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-18T11:52:57.618 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-18T11:52:57.618 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-18T11:52:57.618 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-18T11:52:57.618 INFO:tasks.ceph:Setting up mon.a... 2024-09-18T11:52:57.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-18T11:52:57.651 INFO:teuthology.orchestra.run.smithi154.stdout:creating /etc/ceph/ceph.keyring 2024-09-18T11:52:57.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-18T11:52:57.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T11:52:57.744 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.154'), ('mon.b', '[v2:172.21.15.154:3301,v1:172.21.15.154:6790]'), ('mon.c', '[v2:172.21.15.154:3302,v1:172.21.15.154:6791]')] 2024-09-18T11:52:57.745 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.154,[v2:172.21.15.154:3301,v1:172.21.15.154:6790],[v2:172.21.15.154:3302,v1:172.21.15.154:6791]', 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-18T11:52:57.746 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:52:57.746 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T11:52:57.750 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.154 --addv b '[v2:172.21.15.154:3301,v1:172.21.15.154:6790]' --addv c '[v2:172.21.15.154:3302,v1:172.21.15.154:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:52:57.830 INFO:teuthology.orchestra.run.smithi154.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:52:57.830 INFO:teuthology.orchestra.run.smithi154.stdout:monmaptool: generated fsid c6912087-b15b-4223-a3df-3a06756ee775 2024-09-18T11:52:57.830 INFO:teuthology.orchestra.run.smithi154.stdout:setting min_mon_release = octopus 2024-09-18T11:52:57.830 INFO:teuthology.orchestra.run.smithi154.stdout:epoch 0 2024-09-18T11:52:57.830 INFO:teuthology.orchestra.run.smithi154.stdout:fsid c6912087-b15b-4223-a3df-3a06756ee775 2024-09-18T11:52:57.830 INFO:teuthology.orchestra.run.smithi154.stdout:last_changed 2024-09-18T11:52:57.828924+0000 2024-09-18T11:52:57.830 INFO:teuthology.orchestra.run.smithi154.stdout:created 2024-09-18T11:52:57.828924+0000 2024-09-18T11:52:57.830 INFO:teuthology.orchestra.run.smithi154.stdout:min_mon_release 15 (octopus) 2024-09-18T11:52:57.830 INFO:teuthology.orchestra.run.smithi154.stdout:election_strategy: 1 2024-09-18T11:52:57.830 INFO:teuthology.orchestra.run.smithi154.stdout:0: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.a 2024-09-18T11:52:57.830 INFO:teuthology.orchestra.run.smithi154.stdout:1: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.b 2024-09-18T11:52:57.831 INFO:teuthology.orchestra.run.smithi154.stdout:2: [v2:172.21.15.154:3302/0,v1:172.21.15.154:6791/0] mon.c 2024-09-18T11:52:57.831 INFO:teuthology.orchestra.run.smithi154.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-18T11:52:57.832 DEBUG:teuthology.orchestra.run.smithi154:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T11:52:57.836 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c6912087-b15b-4223-a3df-3a06756ee775... 2024-09-18T11:52:57.838 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T11:52:57.871 INFO:teuthology.orchestra.run.smithi154.stdout:[global] 2024-09-18T11:52:57.871 INFO:teuthology.orchestra.run.smithi154.stdout: chdir = "" 2024-09-18T11:52:57.871 INFO:teuthology.orchestra.run.smithi154.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T11:52:57.871 INFO:teuthology.orchestra.run.smithi154.stdout: auth supported = cephx 2024-09-18T11:52:57.871 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: filestore xattr use omap = true 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: mon clock drift allowed = 1.000 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: osd crush chooseleaf type = 0 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: auth debug = true 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on old message = true 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on bug = true 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: mon pg warn max object skew = 0 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.872 INFO:teuthology.orchestra.run.smithi154.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default size = 2 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow primary affinity = true 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow pg remap = true 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on legacy crush tunables = false 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on crush straw calc version zero = false 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on no sortbitwise = false 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on osd down out interval zero = false 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on too few osds = false 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T11:52:57.873 INFO:teuthology.orchestra.run.smithi154.stdout: mon_allow_pool_size_one = true 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: osd default data pool replay window = 5 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: mon allow pool delete = true 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: mon cluster log file level = debug 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: debug asserts on shutdown = true 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: mon health detail to clog = false 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: mon host = "172.21.15.154,[v2:172.21.15.154:3301,v1:172.21.15.154:6790],[v2:172.21.15.154:3302,v1:172.21.15.154:6791]" 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: mon election default strategy = 1 2024-09-18T11:52:57.874 INFO:teuthology.orchestra.run.smithi154.stdout: ms bind msgr1 = False 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: ms bind msgr2 = True 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: ms type = async 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: fsid = c6912087-b15b-4223-a3df-3a06756ee775 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout:[osd] 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: osd journal size = 100 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub load threshold = 5.0 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub max interval = 600 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: osd mclock profile = high_recovery_ops 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: osd recover clone overlap = true 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: osd recovery max chunk = 1048576 2024-09-18T11:52:57.875 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug shutdown = true 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug op order = true 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify stray on activate = true 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: osd open classes on start = true 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug pg log writeout = true 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: osd deep scrub update digest min age = 30 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: osd map max advance = 10 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: journal zero on create = true 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.876 INFO:teuthology.orchestra.run.smithi154.stdout: filestore ondisk finisher threads = 3 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: filestore apply finisher threads = 3 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: bdev debug aio = true 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug misdirected ops = true 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: bluefs allocator = hybrid 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore allocator = hybrid 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore block size = 96636764160 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore fsck on mount = True 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore zero block detection = True 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluefs = 20 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluestore = 20 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: debug osd = 20 2024-09-18T11:52:57.877 INFO:teuthology.orchestra.run.smithi154.stdout: debug rocksdb = 10 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd full ratio = 0.9 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd nearfull ratio = 0.8 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify cached snaps = True 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify missing on start = True 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: osd failsafe full ratio = 0.95 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: osd mclock override recovery settings = True 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: osd objectstore = bluestore 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: osd op queue = debug_random 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: osd op queue cut off = debug_random 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr] 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: debug mgr = 20 2024-09-18T11:52:57.878 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: mgr/telemetry/nag = false 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout:[mon] 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: debug paxos = 20 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: mon data avail warn = 5 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: mon mgr mkfs grace = 240 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2024-09-18T11:52:57.879 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd reporter subtree level = osd 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd prime pg temp = true 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: # 1m isn't quite enough 2024-09-18T11:52:57.880 INFO:teuthology.orchestra.run.smithi154.stdout: mon_down_mkfs_grace = 2m 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_filestore_osds = false 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout: mon scrub interval = 300 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout:[client] 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout: rgw cache enabled = true 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable ops log = true 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable usage log = true 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.a] 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.b] 2024-09-18T11:52:57.881 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.c] 2024-09-18T11:52:57.882 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-18T11:52:57.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-18T11:52:57.973 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-18T11:52:57.973 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:52:57.973 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-18T11:52:57.978 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:52:57.978 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-18T11:52:57.984 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi154.front.sepia.ceph.com 2024-09-18T11:52:57.984 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:52:57.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T11:52:57.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T11:52:58.007 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:52:58.007 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:52:58.054 INFO:tasks.ceph:Setting up mon nodes... 2024-09-18T11:52:58.054 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-18T11:52:58.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T11:52:58.100 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T11:52:58.103 INFO:tasks.ceph:Setting up mds nodes... 2024-09-18T11:52:58.103 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-18T11:52:58.103 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-18T11:52:58.103 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-18T11:52:58.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-18T11:52:58.115 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T11:52:58.115 INFO:tasks.ceph:role: osd.0 2024-09-18T11:52:58.115 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-18T11:52:58.115 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T11:52:58.637 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:52:58.638 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:52:58.638 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:52:58.638 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 2024-09-18T11:52:58.638 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:52:58.638 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:52:58.638 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:52:58.638 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T11:52:58.638 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:52:58.638 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:52:58.639 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-18T11:52:58.640 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-18T11:52:58.732 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-18T11:52:58.744 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T11:52:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:52:58.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-18T11:52:58.755 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T11:52:58.756 INFO:tasks.ceph:role: osd.1 2024-09-18T11:52:58.756 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-18T11:52:58.756 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T11:52:59.304 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:52:59.304 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:52:59.304 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:52:59.304 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 2024-09-18T11:52:59.305 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:52:59.305 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:52:59.305 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:52:59.305 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T11:52:59.305 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:52:59.305 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:52:59.306 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-18T11:52:59.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-18T11:52:59.347 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-18T11:52:59.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:52:59.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T11:52:59.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-18T11:52:59.365 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T11:52:59.365 INFO:tasks.ceph:role: osd.2 2024-09-18T11:52:59.365 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-18T11:52:59.365 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T11:52:59.893 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T11:52:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T11:52:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T11:52:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 2024-09-18T11:52:59.893 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T11:52:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-18T11:52:59.894 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T11:52:59.894 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T11:52:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T11:52:59.894 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T11:52:59.895 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-18T11:52:59.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-18T11:52:59.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-18T11:52:59.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T11:52:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:52:59.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:53:00.024 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:00.018+0000 7f78603d8540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-18T11:53:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:00.022+0000 7f78603d8540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-18T11:53:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:00.022+0000 7f78603d8540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T11:53:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:00.022+0000 7f78603d8540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T11:53:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:00.022+0000 7f78603d8540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T11:53:00.032 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:00.030+0000 7f78603d8540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-18T11:53:00.598 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:00.594+0000 7f78603d8540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:53:01.382 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:01.378+0000 7f78603d8540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:53:02.176 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:02.174+0000 7f78603d8540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:53:03.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-18T11:53:03.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:53:03.079 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:03.074+0000 7f05c4de6540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-18T11:53:03.080 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:03.074+0000 7f05c4de6540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-18T11:53:03.080 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:03.074+0000 7f05c4de6540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T11:53:03.080 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:03.074+0000 7f05c4de6540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T11:53:03.080 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:03.074+0000 7f05c4de6540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T11:53:03.080 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:03.078+0000 7f05c4de6540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-18T11:53:03.644 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:03.638+0000 7f05c4de6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:53:04.434 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:04.430+0000 7f05c4de6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:53:05.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:05.218+0000 7f05c4de6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:53:06.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-18T11:53:06.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:53:06.163 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:06.158+0000 7f5e741dd540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-18T11:53:06.164 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:06.158+0000 7f5e741dd540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-18T11:53:06.164 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:06.158+0000 7f5e741dd540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T11:53:06.164 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:06.158+0000 7f5e741dd540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T11:53:06.164 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:06.158+0000 7f5e741dd540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T11:53:06.165 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:06.162+0000 7f5e741dd540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-18T11:53:06.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:06.718+0000 7f5e741dd540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:53:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:07.498+0000 7f5e741dd540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:53:08.294 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-18T11:53:08.290+0000 7f5e741dd540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T11:53:09.167 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-18T11:53:09.183 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-18T11:53:09.184 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:53:09.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-18T11:53:09.193 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:53:09.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-18T11:53:09.251 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:53:09.251 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-18T11:53:09.261 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:53:09.261 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-18T11:53:09.276 INFO:tasks.ceph:Adding keys to all mons... 2024-09-18T11:53:09.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T11:53:09.288 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.x] 2024-09-18T11:53:09.288 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAav+pmBavzBRAAH+hql5khe2SqwvNEaK3dTA== 2024-09-18T11:53:09.288 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.0] 2024-09-18T11:53:09.288 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAcv+pmRFlfARAAHOdLdI55G6N0SyFzGC2pZQ== 2024-09-18T11:53:09.288 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.1] 2024-09-18T11:53:09.288 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAfv+pmC4awBBAAju7uEoOO+6x3M1Xl23ASUA== 2024-09-18T11:53:09.288 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.2] 2024-09-18T11:53:09.288 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAiv+pm1Xa1CRAAwLqdJOGEOLhQ1IPMN2EHig== 2024-09-18T11:53:09.289 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T11:53:09.376 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:53:09.415 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:53:09.456 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T11:53:09.544 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-18T11:53:09.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-18T11:53:09.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T11:53:09.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-18T11:53:09.623 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-18T11:53:09.635 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T11:53:09.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-18T11:53:09.699 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-18T11:53:09.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T11:53:09.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-18T11:53:09.816 DEBUG:teuthology.orchestra.run.smithi154:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T11:53:09.820 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-18T11:53:09.821 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T11:53:09.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T11:53:09.866 INFO:tasks.ceph.mon.a:Started 2024-09-18T11:53:09.866 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T11:53:09.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T11:53:09.868 INFO:tasks.ceph.mon.b:Started 2024-09-18T11:53:09.869 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T11:53:09.869 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T11:53:09.871 INFO:tasks.ceph.mon.c:Started 2024-09-18T11:53:09.871 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph config set mgr mgr_pool false --force 2024-09-18T11:53:20.489 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-18T11:53:20.489 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-18T11:53:20.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-18T11:53:20.492 INFO:tasks.ceph.mgr.x:Started 2024-09-18T11:53:20.492 INFO:tasks.ceph:Setting crush tunables to default 2024-09-18T11:53:20.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd crush tunables default 2024-09-18T11:53:20.700 INFO:teuthology.orchestra.run.smithi154.stderr:adjusted tunables profile to default 2024-09-18T11:53:20.710 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-18T11:53:20.710 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:53:20.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-18T11:53:20.720 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:53:20.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-18T11:53:20.733 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-18T11:53:20.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-18T11:53:20.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f 0 2024-09-18T11:53:20.776 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:20.774+0000 7f62bffc3000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-18T11:53:20.944 INFO:teuthology.orchestra.run.smithi154.stdout:0 2024-09-18T11:53:20.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new fd6ca4f9-93c5-4509-8fea-0121afa900f4 1 2024-09-18T11:53:21.155 INFO:teuthology.orchestra.run.smithi154.stdout:1 2024-09-18T11:53:21.167 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 1554e072-019d-4d8e-b2ee-c0aac5a5e811 2 2024-09-18T11:53:21.364 INFO:teuthology.orchestra.run.smithi154.stdout:2 2024-09-18T11:53:21.379 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-18T11:53:21.379 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-18T11:53:21.381 INFO:tasks.ceph.osd.0:Started 2024-09-18T11:53:21.382 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-18T11:53:21.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-18T11:53:21.383 INFO:tasks.ceph.osd.1:Started 2024-09-18T11:53:21.384 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-18T11:53:21.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-18T11:53:21.385 INFO:tasks.ceph.osd.2:Started 2024-09-18T11:53:21.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:53:21.526 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:21.522+0000 7f62bffc3000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-18T11:53:21.562 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:21.562 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":5,"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","created":"2024-09-18T11:53:20.028520+0000","modified":"2024-09-18T11:53:21.358422+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T11:53:21.358422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd6ca4f9-93c5-4509-8fea-0121afa900f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1554e072-019d-4d8e-b2ee-c0aac5a5e811","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:53:21.574 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-18T11:53:21.574 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-18T11:53:21.584 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:21.578+0000 7f62bffc3000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-18T11:53:21.698 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:21.694+0000 7f62bffc3000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-18T11:53:21.782 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-18T11:53:21.778+0000 7fd0cbff7540 -1 Falling back to public interface 2024-09-18T11:53:21.876 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T11:53:21.934 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:21.930+0000 7f62bffc3000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-18T11:53:21.994 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:21.990+0000 7f62bffc3000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-18T11:53:22.013 INFO:tasks.ceph.osd.1.smithi154.stderr:2024-09-18T11:53:22.010+0000 7fd9f5895540 -1 Falling back to public interface 2024-09-18T11:53:22.033 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:53:22.030+0000 7f01956e1540 -1 Falling back to public interface 2024-09-18T11:53:22.103 INFO:teuthology.misc.health.smithi154.stdout: 2024-09-18T11:53:22.103 INFO:teuthology.misc.health.smithi154.stdout:{"epoch":5,"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","created":"2024-09-18T11:53:20.028520+0000","modified":"2024-09-18T11:53:21.358422+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T11:53:21.358422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fd6ca4f9-93c5-4509-8fea-0121afa900f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1554e072-019d-4d8e-b2ee-c0aac5a5e811","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:53:22.113 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-18T11:53:22.224 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:22.222+0000 7f62bffc3000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-18T11:53:22.289 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:22.286+0000 7f62bffc3000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-18T11:53:22.355 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:22.350+0000 7f62bffc3000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-18T11:53:22.810 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:22.806+0000 7f62bffc3000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-18T11:53:22.931 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:22.926+0000 7f62bffc3000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-18T11:53:23.248 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:23.246+0000 7f62bffc3000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-18T11:53:23.414 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:23.410+0000 7f62bffc3000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-18T11:53:23.479 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:23.474+0000 7f62bffc3000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-18T11:53:23.531 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:23.526+0000 7f62bffc3000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-18T11:53:23.588 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:23.582+0000 7f62bffc3000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-18T11:53:23.751 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:23.746+0000 7f62bffc3000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-18T11:53:23.913 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:23.910+0000 7f62bffc3000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-18T11:53:23.978 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:23.974+0000 7f62bffc3000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-18T11:53:24.135 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:24.130+0000 7f62bffc3000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-18T11:53:24.204 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:24.198+0000 7f62bffc3000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-18T11:53:24.974 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:24.970+0000 7f62bffc3000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-18T11:53:25.558 INFO:tasks.ceph.osd.1.smithi154.stderr:2024-09-18T11:53:25.554+0000 7fd9f5895540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:53:25.647 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:53:25.642+0000 7f01956e1540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:53:25.692 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-18T11:53:25.686+0000 7fd0cbff7540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:53:26.025 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:26.022+0000 7f62a9035700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-18T11:53:26.025 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-18T11:53:26.022+0000 7f62a9035700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-18T11:53:27.080 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:53:27.078+0000 7f0190e81700 -1 osd.0 0 waiting for initial osdmap 2024-09-18T11:53:27.087 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-18T11:53:27.082+0000 7fd0c7797700 -1 osd.2 0 waiting for initial osdmap 2024-09-18T11:53:27.087 INFO:tasks.ceph.osd.1.smithi154.stderr:2024-09-18T11:53:27.082+0000 7fd9f285a700 -1 osd.1 0 waiting for initial osdmap 2024-09-18T11:53:27.091 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-18T11:53:27.086+0000 7fd0c053a700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:53:27.091 INFO:tasks.ceph.osd.1.smithi154.stderr:2024-09-18T11:53:27.086+0000 7fd9e9dd8700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:53:27.092 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:53:27.086+0000 7f0189c24700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:53:28.416 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T11:53:28.697 INFO:teuthology.misc.health.smithi154.stdout: 2024-09-18T11:53:28.697 INFO:teuthology.misc.health.smithi154.stdout:{"epoch":8,"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","created":"2024-09-18T11:53:20.028520+0000","modified":"2024-09-18T11:53:28.078673+0000","last_up_change":"2024-09-18T11:53:28.078673+0000","last_in_change":"2024-09-18T11:53:21.358422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":21135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":21135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":21135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":21135}]},"public_addr":"172.21.15.154:6808/21135","cluster_addr":"172.21.15.154:6809/21135","heartbeat_back_addr":"172.21.15.154:6811/21135","heartbeat_front_addr":"172.21.15.154:6810/21135","state":["exists","up"]},{"osd":1,"uuid":"fd6ca4f9-93c5-4509-8fea-0121afa900f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":21134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":21134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":21134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":21134}]},"public_addr":"172.21.15.154:6804/21134","cluster_addr":"172.21.15.154:6805/21134","heartbeat_back_addr":"172.21.15.154:6807/21134","heartbeat_front_addr":"172.21.15.154:6806/21134","state":["exists","up"]},{"osd":2,"uuid":"1554e072-019d-4d8e-b2ee-c0aac5a5e811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":21136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":21136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":21136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":21136}]},"public_addr":"172.21.15.154:6800/21136","cluster_addr":"172.21.15.154:6801/21136","heartbeat_back_addr":"172.21.15.154:6803/21136","heartbeat_front_addr":"172.21.15.154:6802/21136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:53:28.710 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-18T11:53:28.710 INFO:tasks.ceph:Creating RBD pool 2024-09-18T11:53:28.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-18T11:53:29.104 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'rbd' created 2024-09-18T11:53:29.116 DEBUG:teuthology.orchestra.run.smithi154:> rbd --cluster ceph pool init rbd 2024-09-18T11:53:32.122 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-18T11:53:32.123 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-18T11:53:32.123 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-18T11:53:32.123 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-18T11:53:32.124 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-18T11:53:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:32.470 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":4,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":20990}]},"active_addr":"172.21.15.154:6812/20990","active_change":"2024-09-18T11:53:24.993194+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":800381438}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":17256855}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":2325536705}]}]} 2024-09-18T11:53:32.474 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-18T11:53:32.474 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-18T11:53:32.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:53:32.761 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:32.761 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":12,"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","created":"2024-09-18T11:53:20.028520+0000","modified":"2024-09-18T11:53:32.097518+0000","last_up_change":"2024-09-18T11:53:28.078673+0000","last_in_change":"2024-09-18T11:53:21.358422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:53:29.033564+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":21135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":21135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":21135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":21135}]},"public_addr":"172.21.15.154:6808/21135","cluster_addr":"172.21.15.154:6809/21135","heartbeat_back_addr":"172.21.15.154:6811/21135","heartbeat_front_addr":"172.21.15.154:6810/21135","state":["exists","up"]},{"osd":1,"uuid":"fd6ca4f9-93c5-4509-8fea-0121afa900f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":21134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":21134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":21134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":21134}]},"public_addr":"172.21.15.154:6804/21134","cluster_addr":"172.21.15.154:6805/21134","heartbeat_back_addr":"172.21.15.154:6807/21134","heartbeat_front_addr":"172.21.15.154:6806/21134","state":["exists","up"]},{"osd":2,"uuid":"1554e072-019d-4d8e-b2ee-c0aac5a5e811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":21136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":21136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":21136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":21136}]},"public_addr":"172.21.15.154:6800/21136","cluster_addr":"172.21.15.154:6801/21136","heartbeat_back_addr":"172.21.15.154:6803/21136","heartbeat_front_addr":"172.21.15.154:6802/21136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.685684+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.545391+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.679601+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:53:32.781 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-18T11:53:32.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:53:33.070 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:33.070 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":12,"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","created":"2024-09-18T11:53:20.028520+0000","modified":"2024-09-18T11:53:32.097518+0000","last_up_change":"2024-09-18T11:53:28.078673+0000","last_in_change":"2024-09-18T11:53:21.358422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:53:29.033564+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":21135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":21135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":21135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":21135}]},"public_addr":"172.21.15.154:6808/21135","cluster_addr":"172.21.15.154:6809/21135","heartbeat_back_addr":"172.21.15.154:6811/21135","heartbeat_front_addr":"172.21.15.154:6810/21135","state":["exists","up"]},{"osd":1,"uuid":"fd6ca4f9-93c5-4509-8fea-0121afa900f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":21134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":21134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":21134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":21134}]},"public_addr":"172.21.15.154:6804/21134","cluster_addr":"172.21.15.154:6805/21134","heartbeat_back_addr":"172.21.15.154:6807/21134","heartbeat_front_addr":"172.21.15.154:6806/21134","state":["exists","up"]},{"osd":2,"uuid":"1554e072-019d-4d8e-b2ee-c0aac5a5e811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":21136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":21136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":21136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":21136}]},"public_addr":"172.21.15.154:6800/21136","cluster_addr":"172.21.15.154:6801/21136","heartbeat_back_addr":"172.21.15.154:6803/21136","heartbeat_front_addr":"172.21.15.154:6802/21136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.685684+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.545391+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.679601+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:53:33.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:53:33.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T11:53:33.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T11:53:33.225 INFO:teuthology.orchestra.run.smithi154.stdout:34359738371 2024-09-18T11:53:33.225 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:53:33.242 INFO:teuthology.orchestra.run.smithi154.stdout:34359738372 2024-09-18T11:53:33.242 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:53:33.243 INFO:teuthology.orchestra.run.smithi154.stdout:34359738371 2024-09-18T11:53:33.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:53:33.502 INFO:teuthology.orchestra.run.smithi154.stdout:34359738370 2024-09-18T11:53:33.510 INFO:teuthology.orchestra.run.smithi154.stdout:34359738370 2024-09-18T11:53:33.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2024-09-18T11:53:33.517 INFO:teuthology.orchestra.run.smithi154.stdout:34359738371 2024-09-18T11:53:33.523 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2024-09-18T11:53:33.530 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2024-09-18T11:53:34.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:53:34.524 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:53:34.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:53:34.829 INFO:teuthology.orchestra.run.smithi154.stdout:34359738370 2024-09-18T11:53:34.843 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2024-09-18T11:53:34.856 INFO:teuthology.orchestra.run.smithi154.stdout:34359738370 2024-09-18T11:53:34.871 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2024-09-18T11:53:34.874 INFO:teuthology.orchestra.run.smithi154.stdout:34359738371 2024-09-18T11:53:34.891 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2024-09-18T11:53:35.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:53:35.871 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:53:35.891 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:53:36.139 INFO:teuthology.orchestra.run.smithi154.stdout:34359738372 2024-09-18T11:53:36.153 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738372 for osd.2 2024-09-18T11:53:36.154 DEBUG:teuthology.parallel:result is None 2024-09-18T11:53:36.165 INFO:teuthology.orchestra.run.smithi154.stdout:34359738372 2024-09-18T11:53:36.178 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738372 for osd.0 2024-09-18T11:53:36.179 DEBUG:teuthology.parallel:result is None 2024-09-18T11:53:36.213 INFO:teuthology.orchestra.run.smithi154.stdout:34359738373 2024-09-18T11:53:36.226 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.1 2024-09-18T11:53:36.226 DEBUG:teuthology.parallel:result is None 2024-09-18T11:53:36.226 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T11:53:36.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:36.515 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:53:36.525 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-18T11:53:35.012709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":892392,"kb_used_data":320,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282223128,"statfs":{"total":289910292480,"available":288996483072,"internally_reserved":0,"allocated":327680,"data_stored":74000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.108584+0000","last_change":"2024-09-18T11:53:32.108901+0000","last_active":"2024-09-18T11:53:32.108584+0000","last_peered":"2024-09-18T11:53:32.108584+0000","last_clean":"2024-09-18T11:53:32.108584+0000","last_became_active":"2024-09-18T11:53:30.113507+0000","last_became_peered":"2024-09-18T11:53:30.113507+0000","last_unstale":"2024-09-18T11:53:32.108584+0000","last_undegraded":"2024-09-18T11:53:32.108584+0000","last_fullsized":"2024-09-18T11:53:32.108584+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:40:57.342518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.108024+0000","last_change":"2024-09-18T11:53:32.108476+0000","last_active":"2024-09-18T11:53:32.108024+0000","last_peered":"2024-09-18T11:53:32.108024+0000","last_clean":"2024-09-18T11:53:32.108024+0000","last_became_active":"2024-09-18T11:53:30.106924+0000","last_became_peered":"2024-09-18T11:53:30.106924+0000","last_unstale":"2024-09-18T11:53:32.108024+0000","last_undegraded":"2024-09-18T11:53:32.108024+0000","last_fullsized":"2024-09-18T11:53:32.108024+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:01.342017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.596226+0000","last_change":"2024-09-18T11:53:32.596497+0000","last_active":"2024-09-18T11:53:32.596226+0000","last_peered":"2024-09-18T11:53:32.596226+0000","last_clean":"2024-09-18T11:53:32.596226+0000","last_became_active":"2024-09-18T11:53:30.112318+0000","last_became_peered":"2024-09-18T11:53:30.112318+0000","last_unstale":"2024-09-18T11:53:32.596226+0000","last_undegraded":"2024-09-18T11:53:32.596226+0000","last_fullsized":"2024-09-18T11:53:32.596226+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:41:45.444267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047085699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.108120+0000","last_change":"2024-09-18T11:53:32.108453+0000","last_active":"2024-09-18T11:53:32.108120+0000","last_peered":"2024-09-18T11:53:32.108120+0000","last_clean":"2024-09-18T11:53:32.108120+0000","last_became_active":"2024-09-18T11:53:30.106826+0000","last_became_peered":"2024-09-18T11:53:30.106826+0000","last_unstale":"2024-09-18T11:53:32.108120+0000","last_undegraded":"2024-09-18T11:53:32.108120+0000","last_fullsized":"2024-09-18T11:53:32.108120+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:46:21.716863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"10'1","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.107481+0000","last_change":"2024-09-18T11:53:32.107918+0000","last_active":"2024-09-18T11:53:32.107481+0000","last_peered":"2024-09-18T11:53:32.107481+0000","last_clean":"2024-09-18T11:53:32.107481+0000","last_became_active":"2024-09-18T11:53:30.112162+0000","last_became_peered":"2024-09-18T11:53:30.112162+0000","last_unstale":"2024-09-18T11:53:32.107481+0000","last_undegraded":"2024-09-18T11:53:32.107481+0000","last_fullsized":"2024-09-18T11:53:32.107481+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:38:06.385154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"12'2","reported_seq":16,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.107187+0000","last_change":"2024-09-18T11:53:32.107187+0000","last_active":"2024-09-18T11:53:32.107187+0000","last_peered":"2024-09-18T11:53:32.107187+0000","last_clean":"2024-09-18T11:53:32.107187+0000","last_became_active":"2024-09-18T11:53:30.106303+0000","last_became_peered":"2024-09-18T11:53:30.106303+0000","last_unstale":"2024-09-18T11:53:32.107187+0000","last_undegraded":"2024-09-18T11:53:32.107187+0000","last_fullsized":"2024-09-18T11:53:32.107187+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:40:55.788481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.595966+0000","last_change":"2024-09-18T11:53:32.596281+0000","last_active":"2024-09-18T11:53:32.595966+0000","last_peered":"2024-09-18T11:53:32.595966+0000","last_clean":"2024-09-18T11:53:32.595966+0000","last_became_active":"2024-09-18T11:53:30.112377+0000","last_became_peered":"2024-09-18T11:53:30.112377+0000","last_unstale":"2024-09-18T11:53:32.595966+0000","last_undegraded":"2024-09-18T11:53:32.595966+0000","last_fullsized":"2024-09-18T11:53:32.595966+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:55:36.707639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000428744,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.107373+0000","last_change":"2024-09-18T11:53:32.108045+0000","last_active":"2024-09-18T11:53:32.107373+0000","last_peered":"2024-09-18T11:53:32.107373+0000","last_clean":"2024-09-18T11:53:32.107373+0000","last_became_active":"2024-09-18T11:53:30.106253+0000","last_became_peered":"2024-09-18T11:53:30.106253+0000","last_unstale":"2024-09-18T11:53:32.107373+0000","last_undegraded":"2024-09-18T11:53:32.107373+0000","last_fullsized":"2024-09-18T11:53:32.107373+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:40:29.807545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297464,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074376,"statfs":{"total":96636764160,"available":96332161024,"internally_reserved":0,"allocated":106496,"data_stored":24654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297464,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074376,"statfs":{"total":96636764160,"available":96332161024,"internally_reserved":0,"allocated":110592,"data_stored":24673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297464,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074376,"statfs":{"total":96636764160,"available":96332161024,"internally_reserved":0,"allocated":110592,"data_stored":24673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:36.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:36.812 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:36.815 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:53:36.825 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-18T11:53:35.012709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":892392,"kb_used_data":320,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282223128,"statfs":{"total":289910292480,"available":288996483072,"internally_reserved":0,"allocated":327680,"data_stored":74000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.108584+0000","last_change":"2024-09-18T11:53:32.108901+0000","last_active":"2024-09-18T11:53:32.108584+0000","last_peered":"2024-09-18T11:53:32.108584+0000","last_clean":"2024-09-18T11:53:32.108584+0000","last_became_active":"2024-09-18T11:53:30.113507+0000","last_became_peered":"2024-09-18T11:53:30.113507+0000","last_unstale":"2024-09-18T11:53:32.108584+0000","last_undegraded":"2024-09-18T11:53:32.108584+0000","last_fullsized":"2024-09-18T11:53:32.108584+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:40:57.342518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.108024+0000","last_change":"2024-09-18T11:53:32.108476+0000","last_active":"2024-09-18T11:53:32.108024+0000","last_peered":"2024-09-18T11:53:32.108024+0000","last_clean":"2024-09-18T11:53:32.108024+0000","last_became_active":"2024-09-18T11:53:30.106924+0000","last_became_peered":"2024-09-18T11:53:30.106924+0000","last_unstale":"2024-09-18T11:53:32.108024+0000","last_undegraded":"2024-09-18T11:53:32.108024+0000","last_fullsized":"2024-09-18T11:53:32.108024+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:01.342017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.596226+0000","last_change":"2024-09-18T11:53:32.596497+0000","last_active":"2024-09-18T11:53:32.596226+0000","last_peered":"2024-09-18T11:53:32.596226+0000","last_clean":"2024-09-18T11:53:32.596226+0000","last_became_active":"2024-09-18T11:53:30.112318+0000","last_became_peered":"2024-09-18T11:53:30.112318+0000","last_unstale":"2024-09-18T11:53:32.596226+0000","last_undegraded":"2024-09-18T11:53:32.596226+0000","last_fullsized":"2024-09-18T11:53:32.596226+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:41:45.444267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047085699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.108120+0000","last_change":"2024-09-18T11:53:32.108453+0000","last_active":"2024-09-18T11:53:32.108120+0000","last_peered":"2024-09-18T11:53:32.108120+0000","last_clean":"2024-09-18T11:53:32.108120+0000","last_became_active":"2024-09-18T11:53:30.106826+0000","last_became_peered":"2024-09-18T11:53:30.106826+0000","last_unstale":"2024-09-18T11:53:32.108120+0000","last_undegraded":"2024-09-18T11:53:32.108120+0000","last_fullsized":"2024-09-18T11:53:32.108120+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:46:21.716863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"10'1","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.107481+0000","last_change":"2024-09-18T11:53:32.107918+0000","last_active":"2024-09-18T11:53:32.107481+0000","last_peered":"2024-09-18T11:53:32.107481+0000","last_clean":"2024-09-18T11:53:32.107481+0000","last_became_active":"2024-09-18T11:53:30.112162+0000","last_became_peered":"2024-09-18T11:53:30.112162+0000","last_unstale":"2024-09-18T11:53:32.107481+0000","last_undegraded":"2024-09-18T11:53:32.107481+0000","last_fullsized":"2024-09-18T11:53:32.107481+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:38:06.385154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"12'2","reported_seq":16,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.107187+0000","last_change":"2024-09-18T11:53:32.107187+0000","last_active":"2024-09-18T11:53:32.107187+0000","last_peered":"2024-09-18T11:53:32.107187+0000","last_clean":"2024-09-18T11:53:32.107187+0000","last_became_active":"2024-09-18T11:53:30.106303+0000","last_became_peered":"2024-09-18T11:53:30.106303+0000","last_unstale":"2024-09-18T11:53:32.107187+0000","last_undegraded":"2024-09-18T11:53:32.107187+0000","last_fullsized":"2024-09-18T11:53:32.107187+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:40:55.788481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.595966+0000","last_change":"2024-09-18T11:53:32.596281+0000","last_active":"2024-09-18T11:53:32.595966+0000","last_peered":"2024-09-18T11:53:32.595966+0000","last_clean":"2024-09-18T11:53:32.595966+0000","last_became_active":"2024-09-18T11:53:30.112377+0000","last_became_peered":"2024-09-18T11:53:30.112377+0000","last_unstale":"2024-09-18T11:53:32.595966+0000","last_undegraded":"2024-09-18T11:53:32.595966+0000","last_fullsized":"2024-09-18T11:53:32.595966+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:55:36.707639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000428744,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.107373+0000","last_change":"2024-09-18T11:53:32.108045+0000","last_active":"2024-09-18T11:53:32.107373+0000","last_peered":"2024-09-18T11:53:32.107373+0000","last_clean":"2024-09-18T11:53:32.107373+0000","last_became_active":"2024-09-18T11:53:30.106253+0000","last_became_peered":"2024-09-18T11:53:30.106253+0000","last_unstale":"2024-09-18T11:53:32.107373+0000","last_undegraded":"2024-09-18T11:53:32.107373+0000","last_fullsized":"2024-09-18T11:53:32.107373+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:40:29.807545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297464,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074376,"statfs":{"total":96636764160,"available":96332161024,"internally_reserved":0,"allocated":106496,"data_stored":24654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297464,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074376,"statfs":{"total":96636764160,"available":96332161024,"internally_reserved":0,"allocated":110592,"data_stored":24673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297464,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074376,"statfs":{"total":96636764160,"available":96332161024,"internally_reserved":0,"allocated":110592,"data_stored":24673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:36.826 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T11:53:36.826 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-18T11:53:36.826 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-18T11:53:36.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-18T11:53:37.143 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:37.144 INFO:teuthology.orchestra.run.smithi154.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-18T11:53:37.162 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-18T11:53:37.162 INFO:teuthology.run_tasks:Running task rep_lost_unfound_delete... 2024-09-18T11:53:37.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:53:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:37.485 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":12,"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","created":"2024-09-18T11:53:20.028520+0000","modified":"2024-09-18T11:53:32.097518+0000","last_up_change":"2024-09-18T11:53:28.078673+0000","last_in_change":"2024-09-18T11:53:21.358422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:53:29.033564+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":21135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":21135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":21135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":21135}]},"public_addr":"172.21.15.154:6808/21135","cluster_addr":"172.21.15.154:6809/21135","heartbeat_back_addr":"172.21.15.154:6811/21135","heartbeat_front_addr":"172.21.15.154:6810/21135","state":["exists","up"]},{"osd":1,"uuid":"fd6ca4f9-93c5-4509-8fea-0121afa900f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":21134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":21134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":21134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":21134}]},"public_addr":"172.21.15.154:6804/21134","cluster_addr":"172.21.15.154:6805/21134","heartbeat_back_addr":"172.21.15.154:6807/21134","heartbeat_front_addr":"172.21.15.154:6806/21134","state":["exists","up"]},{"osd":2,"uuid":"1554e072-019d-4d8e-b2ee-c0aac5a5e811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":21136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":21136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":21136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":21136}]},"public_addr":"172.21.15.154:6800/21136","cluster_addr":"172.21.15.154:6801/21136","heartbeat_back_addr":"172.21.15.154:6803/21136","heartbeat_front_addr":"172.21.15.154:6802/21136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.685684+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.545391+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.679601+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:53:37.498 INFO:tasks.rep_lost_unfound_delete.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-18T11:53:29.033564+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 12, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2024-09-18T11:53:37.499 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-18T11:53:37.792 INFO:teuthology.orchestra.run.smithi154.stdout:pg_num: 8 2024-09-18T11:53:37.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-18T11:53:38.077 INFO:teuthology.orchestra.run.smithi154.stdout:epoch 12 2024-09-18T11:53:38.077 INFO:teuthology.orchestra.run.smithi154.stdout:fsid c6912087-b15b-4223-a3df-3a06756ee775 2024-09-18T11:53:38.077 INFO:teuthology.orchestra.run.smithi154.stdout:created 2024-09-18T11:53:20.028520+0000 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:modified 2024-09-18T11:53:32.097518+0000 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:crush_version 4 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:full_ratio 0.95 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:backfillfull_ratio 0.9 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:nearfull_ratio 0.85 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:require_min_compat_client luminous 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:min_compat_client jewel 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:require_osd_release quincy 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:stretch_mode_enabled false 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 12 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout: removed_snaps_queue [2~1] 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:max_osd 3 2024-09-18T11:53:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6808/21135 v2:172.21.15.154:6809/21135 exists,up db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f 2024-09-18T11:53:38.079 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6804/21134 v2:172.21.15.154:6805/21134 exists,up fd6ca4f9-93c5-4509-8fea-0121afa900f4 2024-09-18T11:53:38.079 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6800/21136 v2:172.21.15.154:6801/21136 exists,up 1554e072-019d-4d8e-b2ee-c0aac5a5e811 2024-09-18T11:53:38.091 INFO:tasks.rep_lost_unfound_delete.ceph_manager:['osd.0 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6808/21135 v2:172.21.15.154:6809/21135 exists,up db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f', 'osd.1 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6804/21134 v2:172.21.15.154:6805/21134 exists,up fd6ca4f9-93c5-4509-8fea-0121afa900f4', 'osd.2 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6800/21136 v2:172.21.15.154:6801/21136 exists,up 1554e072-019d-4d8e-b2ee-c0aac5a5e811'] 2024-09-18T11:53:38.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:53:38.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T11:53:38.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T11:53:38.228 INFO:teuthology.orchestra.run.smithi154.stdout:34359738373 2024-09-18T11:53:38.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:53:38.230 INFO:teuthology.orchestra.run.smithi154.stdout:34359738374 2024-09-18T11:53:38.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:53:38.238 INFO:teuthology.orchestra.run.smithi154.stdout:34359738373 2024-09-18T11:53:38.238 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:53:38.514 INFO:teuthology.orchestra.run.smithi154.stdout:34359738372 2024-09-18T11:53:38.532 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738372 for osd.0 2024-09-18T11:53:38.532 INFO:teuthology.orchestra.run.smithi154.stdout:34359738373 2024-09-18T11:53:38.544 INFO:teuthology.orchestra.run.smithi154.stdout:34359738372 2024-09-18T11:53:38.549 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738374 got 34359738373 for osd.1 2024-09-18T11:53:38.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738372 for osd.2 2024-09-18T11:53:39.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:53:39.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:53:39.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:53:39.832 INFO:teuthology.orchestra.run.smithi154.stdout:34359738373 2024-09-18T11:53:39.849 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738373 for osd.0 2024-09-18T11:53:39.849 DEBUG:teuthology.parallel:result is None 2024-09-18T11:53:39.864 INFO:teuthology.orchestra.run.smithi154.stdout:34359738373 2024-09-18T11:53:39.876 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738373 for osd.2 2024-09-18T11:53:39.876 DEBUG:teuthology.parallel:result is None 2024-09-18T11:53:39.878 INFO:teuthology.orchestra.run.smithi154.stdout:34359738374 2024-09-18T11:53:39.891 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738374 got 34359738374 for osd.1 2024-09-18T11:53:39.891 DEBUG:teuthology.parallel:result is None 2024-09-18T11:53:39.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for clean 2024-09-18T11:53:39.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:40.174 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:40.177 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:53:40.187 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-18T11:53:39.953060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":892392,"kb_used_data":320,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282223128,"statfs":{"total":289910292480,"available":288996483072,"internally_reserved":0,"allocated":327680,"data_stored":74000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.940351"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.108584+0000","last_change":"2024-09-18T11:53:32.108901+0000","last_active":"2024-09-18T11:53:32.108584+0000","last_peered":"2024-09-18T11:53:32.108584+0000","last_clean":"2024-09-18T11:53:32.108584+0000","last_became_active":"2024-09-18T11:53:30.113507+0000","last_became_peered":"2024-09-18T11:53:30.113507+0000","last_unstale":"2024-09-18T11:53:32.108584+0000","last_undegraded":"2024-09-18T11:53:32.108584+0000","last_fullsized":"2024-09-18T11:53:32.108584+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:40:57.342518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.108024+0000","last_change":"2024-09-18T11:53:32.108476+0000","last_active":"2024-09-18T11:53:32.108024+0000","last_peered":"2024-09-18T11:53:32.108024+0000","last_clean":"2024-09-18T11:53:32.108024+0000","last_became_active":"2024-09-18T11:53:30.106924+0000","last_became_peered":"2024-09-18T11:53:30.106924+0000","last_unstale":"2024-09-18T11:53:32.108024+0000","last_undegraded":"2024-09-18T11:53:32.108024+0000","last_fullsized":"2024-09-18T11:53:32.108024+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:01.342017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.596226+0000","last_change":"2024-09-18T11:53:32.596497+0000","last_active":"2024-09-18T11:53:32.596226+0000","last_peered":"2024-09-18T11:53:32.596226+0000","last_clean":"2024-09-18T11:53:32.596226+0000","last_became_active":"2024-09-18T11:53:30.112318+0000","last_became_peered":"2024-09-18T11:53:30.112318+0000","last_unstale":"2024-09-18T11:53:32.596226+0000","last_undegraded":"2024-09-18T11:53:32.596226+0000","last_fullsized":"2024-09-18T11:53:32.596226+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:41:45.444267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047085699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.108120+0000","last_change":"2024-09-18T11:53:32.108453+0000","last_active":"2024-09-18T11:53:32.108120+0000","last_peered":"2024-09-18T11:53:32.108120+0000","last_clean":"2024-09-18T11:53:32.108120+0000","last_became_active":"2024-09-18T11:53:30.106826+0000","last_became_peered":"2024-09-18T11:53:30.106826+0000","last_unstale":"2024-09-18T11:53:32.108120+0000","last_undegraded":"2024-09-18T11:53:32.108120+0000","last_fullsized":"2024-09-18T11:53:32.108120+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:46:21.716863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"10'1","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.107481+0000","last_change":"2024-09-18T11:53:32.107918+0000","last_active":"2024-09-18T11:53:32.107481+0000","last_peered":"2024-09-18T11:53:32.107481+0000","last_clean":"2024-09-18T11:53:32.107481+0000","last_became_active":"2024-09-18T11:53:30.112162+0000","last_became_peered":"2024-09-18T11:53:30.112162+0000","last_unstale":"2024-09-18T11:53:32.107481+0000","last_undegraded":"2024-09-18T11:53:32.107481+0000","last_fullsized":"2024-09-18T11:53:32.107481+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:38:06.385154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"12'2","reported_seq":16,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.107187+0000","last_change":"2024-09-18T11:53:32.107187+0000","last_active":"2024-09-18T11:53:32.107187+0000","last_peered":"2024-09-18T11:53:32.107187+0000","last_clean":"2024-09-18T11:53:32.107187+0000","last_became_active":"2024-09-18T11:53:30.106303+0000","last_became_peered":"2024-09-18T11:53:30.106303+0000","last_unstale":"2024-09-18T11:53:32.107187+0000","last_undegraded":"2024-09-18T11:53:32.107187+0000","last_fullsized":"2024-09-18T11:53:32.107187+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:40:55.788481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.595966+0000","last_change":"2024-09-18T11:53:32.596281+0000","last_active":"2024-09-18T11:53:32.595966+0000","last_peered":"2024-09-18T11:53:32.595966+0000","last_clean":"2024-09-18T11:53:32.595966+0000","last_became_active":"2024-09-18T11:53:30.112377+0000","last_became_peered":"2024-09-18T11:53:30.112377+0000","last_unstale":"2024-09-18T11:53:32.595966+0000","last_undegraded":"2024-09-18T11:53:32.595966+0000","last_fullsized":"2024-09-18T11:53:32.595966+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:55:36.707639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000428744,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.107373+0000","last_change":"2024-09-18T11:53:32.108045+0000","last_active":"2024-09-18T11:53:32.107373+0000","last_peered":"2024-09-18T11:53:32.107373+0000","last_clean":"2024-09-18T11:53:32.107373+0000","last_became_active":"2024-09-18T11:53:30.106253+0000","last_became_peered":"2024-09-18T11:53:30.106253+0000","last_unstale":"2024-09-18T11:53:32.107373+0000","last_undegraded":"2024-09-18T11:53:32.107373+0000","last_fullsized":"2024-09-18T11:53:32.107373+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:40:29.807545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297464,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074376,"statfs":{"total":96636764160,"available":96332161024,"internally_reserved":0,"allocated":106496,"data_stored":24654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297464,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074376,"statfs":{"total":96636764160,"available":96332161024,"internally_reserved":0,"allocated":110592,"data_stored":24673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297464,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074376,"statfs":{"total":96636764160,"available":96332161024,"internally_reserved":0,"allocated":110592,"data_stored":24673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:40.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:40.471 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:40.474 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:53:40.484 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-18T11:53:39.953060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":892392,"kb_used_data":320,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282223128,"statfs":{"total":289910292480,"available":288996483072,"internally_reserved":0,"allocated":327680,"data_stored":74000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.940351"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.108584+0000","last_change":"2024-09-18T11:53:32.108901+0000","last_active":"2024-09-18T11:53:32.108584+0000","last_peered":"2024-09-18T11:53:32.108584+0000","last_clean":"2024-09-18T11:53:32.108584+0000","last_became_active":"2024-09-18T11:53:30.113507+0000","last_became_peered":"2024-09-18T11:53:30.113507+0000","last_unstale":"2024-09-18T11:53:32.108584+0000","last_undegraded":"2024-09-18T11:53:32.108584+0000","last_fullsized":"2024-09-18T11:53:32.108584+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:40:57.342518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.108024+0000","last_change":"2024-09-18T11:53:32.108476+0000","last_active":"2024-09-18T11:53:32.108024+0000","last_peered":"2024-09-18T11:53:32.108024+0000","last_clean":"2024-09-18T11:53:32.108024+0000","last_became_active":"2024-09-18T11:53:30.106924+0000","last_became_peered":"2024-09-18T11:53:30.106924+0000","last_unstale":"2024-09-18T11:53:32.108024+0000","last_undegraded":"2024-09-18T11:53:32.108024+0000","last_fullsized":"2024-09-18T11:53:32.108024+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:01.342017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.596226+0000","last_change":"2024-09-18T11:53:32.596497+0000","last_active":"2024-09-18T11:53:32.596226+0000","last_peered":"2024-09-18T11:53:32.596226+0000","last_clean":"2024-09-18T11:53:32.596226+0000","last_became_active":"2024-09-18T11:53:30.112318+0000","last_became_peered":"2024-09-18T11:53:30.112318+0000","last_unstale":"2024-09-18T11:53:32.596226+0000","last_undegraded":"2024-09-18T11:53:32.596226+0000","last_fullsized":"2024-09-18T11:53:32.596226+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:41:45.444267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047085699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.108120+0000","last_change":"2024-09-18T11:53:32.108453+0000","last_active":"2024-09-18T11:53:32.108120+0000","last_peered":"2024-09-18T11:53:32.108120+0000","last_clean":"2024-09-18T11:53:32.108120+0000","last_became_active":"2024-09-18T11:53:30.106826+0000","last_became_peered":"2024-09-18T11:53:30.106826+0000","last_unstale":"2024-09-18T11:53:32.108120+0000","last_undegraded":"2024-09-18T11:53:32.108120+0000","last_fullsized":"2024-09-18T11:53:32.108120+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:46:21.716863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"10'1","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.107481+0000","last_change":"2024-09-18T11:53:32.107918+0000","last_active":"2024-09-18T11:53:32.107481+0000","last_peered":"2024-09-18T11:53:32.107481+0000","last_clean":"2024-09-18T11:53:32.107481+0000","last_became_active":"2024-09-18T11:53:30.112162+0000","last_became_peered":"2024-09-18T11:53:30.112162+0000","last_unstale":"2024-09-18T11:53:32.107481+0000","last_undegraded":"2024-09-18T11:53:32.107481+0000","last_fullsized":"2024-09-18T11:53:32.107481+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:38:06.385154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"12'2","reported_seq":16,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.107187+0000","last_change":"2024-09-18T11:53:32.107187+0000","last_active":"2024-09-18T11:53:32.107187+0000","last_peered":"2024-09-18T11:53:32.107187+0000","last_clean":"2024-09-18T11:53:32.107187+0000","last_became_active":"2024-09-18T11:53:30.106303+0000","last_became_peered":"2024-09-18T11:53:30.106303+0000","last_unstale":"2024-09-18T11:53:32.107187+0000","last_undegraded":"2024-09-18T11:53:32.107187+0000","last_fullsized":"2024-09-18T11:53:32.107187+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:40:55.788481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.595966+0000","last_change":"2024-09-18T11:53:32.596281+0000","last_active":"2024-09-18T11:53:32.595966+0000","last_peered":"2024-09-18T11:53:32.595966+0000","last_clean":"2024-09-18T11:53:32.595966+0000","last_became_active":"2024-09-18T11:53:30.112377+0000","last_became_peered":"2024-09-18T11:53:30.112377+0000","last_unstale":"2024-09-18T11:53:32.595966+0000","last_undegraded":"2024-09-18T11:53:32.595966+0000","last_fullsized":"2024-09-18T11:53:32.595966+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:55:36.707639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000428744,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-18T11:53:32.107373+0000","last_change":"2024-09-18T11:53:32.108045+0000","last_active":"2024-09-18T11:53:32.107373+0000","last_peered":"2024-09-18T11:53:32.107373+0000","last_clean":"2024-09-18T11:53:32.107373+0000","last_became_active":"2024-09-18T11:53:30.106253+0000","last_became_peered":"2024-09-18T11:53:30.106253+0000","last_unstale":"2024-09-18T11:53:32.107373+0000","last_undegraded":"2024-09-18T11:53:32.107373+0000","last_fullsized":"2024-09-18T11:53:32.107373+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:40:29.807545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297464,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074376,"statfs":{"total":96636764160,"available":96332161024,"internally_reserved":0,"allocated":106496,"data_stored":24654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297464,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074376,"statfs":{"total":96636764160,"available":96332161024,"internally_reserved":0,"allocated":110592,"data_stored":24673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297464,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074376,"statfs":{"total":96636764160,"available":96332161024,"internally_reserved":0,"allocated":110592,"data_stored":24673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:40.485 INFO:tasks.rep_lost_unfound_delete.ceph_manager:clean! 2024-09-18T11:53:40.485 INFO:tasks.rep_lost_unfound_delete.ceph_manager:creating pool_name unfounddel_pool 2024-09-18T11:53:40.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unfounddel_pool 16 2024-09-18T11:53:41.203 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unfounddel_pool' created 2024-09-18T11:53:41.213 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unfounddel_pool rados --yes-i-really-mean-it || true 2024-09-18T11:53:42.217 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rados' on pool 'unfounddel_pool' 2024-09-18T11:53:43.228 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-18T11:53:43.229 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:53:43.303 INFO:tasks.ceph.osd.2:Stopped 2024-09-18T11:53:43.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 2 2024-09-18T11:53:43.606 INFO:tasks.ceph.mon.a.smithi154.stderr:2024-09-18T11:53:43.601+0000 7f2c7dcba700 -1 mon.a@0(leader).osd e16 definitely_dead 0 2024-09-18T11:53:44.233 INFO:teuthology.orchestra.run.smithi154.stderr:marked down osd.2. 2024-09-18T11:53:44.245 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2024-09-18T11:53:44.540 INFO:tasks.ceph.mon.a.smithi154.stderr:2024-09-18T11:53:44.533+0000 7f2c7dcba700 -1 mon.a@0(leader).osd e17 definitely_dead 0 2024-09-18T11:53:45.242 INFO:teuthology.orchestra.run.smithi154.stderr:marked out osd.2. 2024-09-18T11:53:45.254 INFO:tasks.util.rados:rados -p unfounddel_pool put dummy /etc/fstab 2024-09-18T11:53:45.254 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put dummy /etc/fstab 2024-09-18T11:53:45.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:53:45.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T11:53:45.439 INFO:teuthology.orchestra.run.smithi154.stdout:34359738377 2024-09-18T11:53:45.440 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:53:45.443 INFO:teuthology.orchestra.run.smithi154.stdout:34359738376 2024-09-18T11:53:45.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:53:45.734 INFO:teuthology.orchestra.run.smithi154.stdout:34359738376 2024-09-18T11:53:45.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738377 got 34359738376 for osd.1 2024-09-18T11:53:45.769 INFO:teuthology.orchestra.run.smithi154.stdout:34359738375 2024-09-18T11:53:45.789 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738376 got 34359738375 for osd.0 2024-09-18T11:53:46.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:53:46.790 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:53:47.045 INFO:teuthology.orchestra.run.smithi154.stdout:34359738376 2024-09-18T11:53:47.065 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738377 got 34359738376 for osd.1 2024-09-18T11:53:47.069 INFO:teuthology.orchestra.run.smithi154.stdout:34359738375 2024-09-18T11:53:47.082 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738376 got 34359738375 for osd.0 2024-09-18T11:53:48.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:53:48.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:53:48.380 INFO:teuthology.orchestra.run.smithi154.stdout:34359738377 2024-09-18T11:53:48.393 INFO:teuthology.orchestra.run.smithi154.stdout:34359738376 2024-09-18T11:53:48.393 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738377 got 34359738377 for osd.1 2024-09-18T11:53:48.394 DEBUG:teuthology.parallel:result is None 2024-09-18T11:53:48.410 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738376 got 34359738376 for osd.0 2024-09-18T11:53:48.410 DEBUG:teuthology.parallel:result is None 2024-09-18T11:53:48.411 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for recovery to complete 2024-09-18T11:53:48.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:48.699 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:48.702 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:53:48.713 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-18T11:53:47.015813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":636,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595024,"kb_used_data":308,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148656,"statfs":{"total":193273528320,"available":192664223744,"internally_reserved":0,"allocated":315392,"data_stored":81744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001028"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.230105+0000","last_change":"2024-09-18T11:53:42.210495+0000","last_active":"2024-09-18T11:53:44.230105+0000","last_peered":"2024-09-18T11:53:44.230105+0000","last_clean":"2024-09-18T11:53:44.230105+0000","last_became_active":"2024-09-18T11:53:42.210220+0000","last_became_peered":"2024-09-18T11:53:42.210220+0000","last_unstale":"2024-09-18T11:53:44.230105+0000","last_undegraded":"2024-09-18T11:53:44.230105+0000","last_fullsized":"2024-09-18T11:53:44.230105+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:15:30.898270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.235593+0000","last_change":"2024-09-18T11:53:45.235960+0000","last_active":"2024-09-18T11:53:41.166841+0000","last_peered":"2024-09-18T11:53:41.166841+0000","last_clean":"2024-09-18T11:53:41.166841+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.235593+0000","last_undegraded":"2024-09-18T11:53:45.235593+0000","last_fullsized":"2024-09-18T11:53:45.235593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:33:09.990682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.238821+0000","last_change":"2024-09-18T11:53:45.239008+0000","last_active":"2024-09-18T11:53:44.226410+0000","last_peered":"2024-09-18T11:53:43.212931+0000","last_clean":"2024-09-18T11:53:43.212931+0000","last_became_active":"2024-09-18T11:53:42.213226+0000","last_became_peered":"2024-09-18T11:53:42.213226+0000","last_unstale":"2024-09-18T11:53:45.238821+0000","last_undegraded":"2024-09-18T11:53:45.238821+0000","last_fullsized":"2024-09-18T11:53:45.238821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:10.557607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.233727+0000","last_change":"2024-09-18T11:53:45.234102+0000","last_active":"2024-09-18T11:53:41.166841+0000","last_peered":"2024-09-18T11:53:41.166841+0000","last_clean":"2024-09-18T11:53:41.166841+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.233727+0000","last_undegraded":"2024-09-18T11:53:45.233727+0000","last_fullsized":"2024-09-18T11:53:45.233727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:38:07.951416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.234657+0000","last_change":"2024-09-18T11:53:45.235027+0000","last_active":"2024-09-18T11:53:44.227135+0000","last_peered":"2024-09-18T11:53:43.208421+0000","last_clean":"2024-09-18T11:53:43.208421+0000","last_became_active":"2024-09-18T11:53:42.203696+0000","last_became_peered":"2024-09-18T11:53:42.203696+0000","last_unstale":"2024-09-18T11:53:45.234657+0000","last_undegraded":"2024-09-18T11:53:45.234657+0000","last_fullsized":"2024-09-18T11:53:45.234657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:50.811957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.225199+0000","last_change":"2024-09-18T11:53:42.212949+0000","last_active":"2024-09-18T11:53:44.225199+0000","last_peered":"2024-09-18T11:53:44.225199+0000","last_clean":"2024-09-18T11:53:44.225199+0000","last_became_active":"2024-09-18T11:53:42.211869+0000","last_became_peered":"2024-09-18T11:53:42.211869+0000","last_unstale":"2024-09-18T11:53:44.225199+0000","last_undegraded":"2024-09-18T11:53:44.225199+0000","last_fullsized":"2024-09-18T11:53:44.225199+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:34:23.038561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.227190+0000","last_change":"2024-09-18T11:53:42.209939+0000","last_active":"2024-09-18T11:53:44.227190+0000","last_peered":"2024-09-18T11:53:44.227190+0000","last_clean":"2024-09-18T11:53:44.227190+0000","last_became_active":"2024-09-18T11:53:42.209659+0000","last_became_peered":"2024-09-18T11:53:42.209659+0000","last_unstale":"2024-09-18T11:53:44.227190+0000","last_undegraded":"2024-09-18T11:53:44.227190+0000","last_fullsized":"2024-09-18T11:53:44.227190+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:24:24.722802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.233066+0000","last_change":"2024-09-18T11:53:45.233457+0000","last_active":"2024-09-18T11:53:44.225279+0000","last_peered":"2024-09-18T11:53:43.208142+0000","last_clean":"2024-09-18T11:53:43.208142+0000","last_became_active":"2024-09-18T11:53:42.203492+0000","last_became_peered":"2024-09-18T11:53:42.203492+0000","last_unstale":"2024-09-18T11:53:45.233066+0000","last_undegraded":"2024-09-18T11:53:45.233066+0000","last_fullsized":"2024-09-18T11:53:45.233066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:47:45.902672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.225007+0000","last_change":"2024-09-18T11:53:32.108045+0000","last_active":"2024-09-18T11:53:44.225007+0000","last_peered":"2024-09-18T11:53:44.225007+0000","last_clean":"2024-09-18T11:53:44.225007+0000","last_became_active":"2024-09-18T11:53:30.106253+0000","last_became_peered":"2024-09-18T11:53:30.106253+0000","last_unstale":"2024-09-18T11:53:44.225007+0000","last_undegraded":"2024-09-18T11:53:44.225007+0000","last_fullsized":"2024-09-18T11:53:44.225007+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:40:29.807545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.238702+0000","last_change":"2024-09-18T11:53:45.238922+0000","last_active":"2024-09-18T11:53:44.225949+0000","last_peered":"2024-09-18T11:53:43.212902+0000","last_clean":"2024-09-18T11:53:43.212902+0000","last_became_active":"2024-09-18T11:53:42.213577+0000","last_became_peered":"2024-09-18T11:53:42.213577+0000","last_unstale":"2024-09-18T11:53:45.238702+0000","last_undegraded":"2024-09-18T11:53:45.238702+0000","last_fullsized":"2024-09-18T11:53:45.238702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:55:55.551044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.233339+0000","last_change":"2024-09-18T11:53:45.233684+0000","last_active":"2024-09-18T11:53:29.091408+0000","last_peered":"2024-09-18T11:53:29.091408+0000","last_clean":"2024-09-18T11:53:29.091408+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.233339+0000","last_undegraded":"2024-09-18T11:53:45.233339+0000","last_fullsized":"2024-09-18T11:53:45.233339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:39:35.340135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.227034+0000","last_change":"2024-09-18T11:53:42.209716+0000","last_active":"2024-09-18T11:53:44.227034+0000","last_peered":"2024-09-18T11:53:44.227034+0000","last_clean":"2024-09-18T11:53:44.227034+0000","last_became_active":"2024-09-18T11:53:42.209437+0000","last_became_peered":"2024-09-18T11:53:42.209437+0000","last_unstale":"2024-09-18T11:53:44.227034+0000","last_undegraded":"2024-09-18T11:53:44.227034+0000","last_fullsized":"2024-09-18T11:53:44.227034+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:16:27.733240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":23,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.227203+0000","last_change":"2024-09-18T11:53:32.107187+0000","last_active":"2024-09-18T11:53:44.227203+0000","last_peered":"2024-09-18T11:53:44.227203+0000","last_clean":"2024-09-18T11:53:44.227203+0000","last_became_active":"2024-09-18T11:53:30.106303+0000","last_became_peered":"2024-09-18T11:53:30.106303+0000","last_unstale":"2024-09-18T11:53:44.227203+0000","last_undegraded":"2024-09-18T11:53:44.227203+0000","last_fullsized":"2024-09-18T11:53:44.227203+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:40:55.788481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.238217+0000","last_change":"2024-09-18T11:53:45.238639+0000","last_active":"2024-09-18T11:53:41.166841+0000","last_peered":"2024-09-18T11:53:41.166841+0000","last_clean":"2024-09-18T11:53:41.166841+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.238217+0000","last_undegraded":"2024-09-18T11:53:45.238217+0000","last_fullsized":"2024-09-18T11:53:45.238217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:01:45.711737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":26,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.239936+0000","last_change":"2024-09-18T11:53:45.240129+0000","last_active":"2024-09-18T11:53:44.226197+0000","last_peered":"2024-09-18T11:53:43.213133+0000","last_clean":"2024-09-18T11:53:43.213133+0000","last_became_active":"2024-09-18T11:53:30.112162+0000","last_became_peered":"2024-09-18T11:53:30.112162+0000","last_unstale":"2024-09-18T11:53:45.239936+0000","last_undegraded":"2024-09-18T11:53:45.239936+0000","last_fullsized":"2024-09-18T11:53:45.239936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:13:49.671400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.237648+0000","last_change":"2024-09-18T11:53:45.237972+0000","last_active":"2024-09-18T11:53:41.166841+0000","last_peered":"2024-09-18T11:53:41.166841+0000","last_clean":"2024-09-18T11:53:41.166841+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.237648+0000","last_undegraded":"2024-09-18T11:53:45.237648+0000","last_fullsized":"2024-09-18T11:53:45.237648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:38:09.530322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.227413+0000","last_change":"2024-09-18T11:53:42.213087+0000","last_active":"2024-09-18T11:53:44.227413+0000","last_peered":"2024-09-18T11:53:44.227413+0000","last_clean":"2024-09-18T11:53:44.227413+0000","last_became_active":"2024-09-18T11:53:42.211893+0000","last_became_peered":"2024-09-18T11:53:42.211893+0000","last_unstale":"2024-09-18T11:53:44.227413+0000","last_undegraded":"2024-09-18T11:53:44.227413+0000","last_fullsized":"2024-09-18T11:53:44.227413+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:46:15.719065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.225550+0000","last_change":"2024-09-18T11:53:32.108453+0000","last_active":"2024-09-18T11:53:44.225550+0000","last_peered":"2024-09-18T11:53:44.225550+0000","last_clean":"2024-09-18T11:53:44.225550+0000","last_became_active":"2024-09-18T11:53:30.106826+0000","last_became_peered":"2024-09-18T11:53:30.106826+0000","last_unstale":"2024-09-18T11:53:44.225550+0000","last_undegraded":"2024-09-18T11:53:44.225550+0000","last_fullsized":"2024-09-18T11:53:44.225550+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:46:21.716863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.226775+0000","last_change":"2024-09-18T11:53:42.210828+0000","last_active":"2024-09-18T11:53:44.226775+0000","last_peered":"2024-09-18T11:53:44.226775+0000","last_clean":"2024-09-18T11:53:44.226775+0000","last_became_active":"2024-09-18T11:53:42.210556+0000","last_became_peered":"2024-09-18T11:53:42.210556+0000","last_unstale":"2024-09-18T11:53:44.226775+0000","last_undegraded":"2024-09-18T11:53:44.226775+0000","last_fullsized":"2024-09-18T11:53:44.226775+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:39:13.072068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.235177+0000","last_change":"2024-09-18T11:53:45.235556+0000","last_active":"2024-09-18T11:53:29.091408+0000","last_peered":"2024-09-18T11:53:29.091408+0000","last_clean":"2024-09-18T11:53:29.091408+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.235177+0000","last_undegraded":"2024-09-18T11:53:45.235177+0000","last_fullsized":"2024-09-18T11:53:45.235177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:07:13.898948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.226611+0000","last_change":"2024-09-18T11:53:42.212946+0000","last_active":"2024-09-18T11:53:44.226611+0000","last_peered":"2024-09-18T11:53:44.226611+0000","last_clean":"2024-09-18T11:53:44.226611+0000","last_became_active":"2024-09-18T11:53:42.211801+0000","last_became_peered":"2024-09-18T11:53:42.211801+0000","last_unstale":"2024-09-18T11:53:44.226611+0000","last_undegraded":"2024-09-18T11:53:44.226611+0000","last_fullsized":"2024-09-18T11:53:44.226611+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:39:50.787931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.226920+0000","last_change":"2024-09-18T11:53:32.108476+0000","last_active":"2024-09-18T11:53:44.226920+0000","last_peered":"2024-09-18T11:53:44.226920+0000","last_clean":"2024-09-18T11:53:44.226920+0000","last_became_active":"2024-09-18T11:53:30.106924+0000","last_became_peered":"2024-09-18T11:53:30.106924+0000","last_unstale":"2024-09-18T11:53:44.226920+0000","last_undegraded":"2024-09-18T11:53:44.226920+0000","last_fullsized":"2024-09-18T11:53:44.226920+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:01.342017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":25,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.240035+0000","last_change":"2024-09-18T11:53:45.240199+0000","last_active":"2024-09-18T11:53:44.228107+0000","last_peered":"2024-09-18T11:53:43.213397+0000","last_clean":"2024-09-18T11:53:43.213397+0000","last_became_active":"2024-09-18T11:53:30.113507+0000","last_became_peered":"2024-09-18T11:53:30.113507+0000","last_unstale":"2024-09-18T11:53:45.240035+0000","last_undegraded":"2024-09-18T11:53:45.240035+0000","last_fullsized":"2024-09-18T11:53:45.240035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:50.421441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"18'1","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.238066+0000","last_change":"2024-09-18T11:53:42.209989+0000","last_active":"2024-09-18T11:53:45.238066+0000","last_peered":"2024-09-18T11:53:45.238066+0000","last_clean":"2024-09-18T11:53:45.238066+0000","last_became_active":"2024-09-18T11:53:42.209680+0000","last_became_peered":"2024-09-18T11:53:42.209680+0000","last_unstale":"2024-09-18T11:53:45.238066+0000","last_undegraded":"2024-09-18T11:53:45.238066+0000","last_fullsized":"2024-09-18T11:53:45.238066+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:52:12.355930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297520,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074320,"statfs":{"total":96636764160,"available":96332103680,"internally_reserved":0,"allocated":163840,"data_stored":42562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":151552,"data_stored":39182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:48.714 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:49.001 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:49.003 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:53:49.018 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-18T11:53:47.015813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":636,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595024,"kb_used_data":308,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148656,"statfs":{"total":193273528320,"available":192664223744,"internally_reserved":0,"allocated":315392,"data_stored":81744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001028"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.230105+0000","last_change":"2024-09-18T11:53:42.210495+0000","last_active":"2024-09-18T11:53:44.230105+0000","last_peered":"2024-09-18T11:53:44.230105+0000","last_clean":"2024-09-18T11:53:44.230105+0000","last_became_active":"2024-09-18T11:53:42.210220+0000","last_became_peered":"2024-09-18T11:53:42.210220+0000","last_unstale":"2024-09-18T11:53:44.230105+0000","last_undegraded":"2024-09-18T11:53:44.230105+0000","last_fullsized":"2024-09-18T11:53:44.230105+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:15:30.898270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.235593+0000","last_change":"2024-09-18T11:53:45.235960+0000","last_active":"2024-09-18T11:53:41.166841+0000","last_peered":"2024-09-18T11:53:41.166841+0000","last_clean":"2024-09-18T11:53:41.166841+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.235593+0000","last_undegraded":"2024-09-18T11:53:45.235593+0000","last_fullsized":"2024-09-18T11:53:45.235593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:33:09.990682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.238821+0000","last_change":"2024-09-18T11:53:45.239008+0000","last_active":"2024-09-18T11:53:44.226410+0000","last_peered":"2024-09-18T11:53:43.212931+0000","last_clean":"2024-09-18T11:53:43.212931+0000","last_became_active":"2024-09-18T11:53:42.213226+0000","last_became_peered":"2024-09-18T11:53:42.213226+0000","last_unstale":"2024-09-18T11:53:45.238821+0000","last_undegraded":"2024-09-18T11:53:45.238821+0000","last_fullsized":"2024-09-18T11:53:45.238821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:10.557607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.233727+0000","last_change":"2024-09-18T11:53:45.234102+0000","last_active":"2024-09-18T11:53:41.166841+0000","last_peered":"2024-09-18T11:53:41.166841+0000","last_clean":"2024-09-18T11:53:41.166841+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.233727+0000","last_undegraded":"2024-09-18T11:53:45.233727+0000","last_fullsized":"2024-09-18T11:53:45.233727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:38:07.951416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.234657+0000","last_change":"2024-09-18T11:53:45.235027+0000","last_active":"2024-09-18T11:53:44.227135+0000","last_peered":"2024-09-18T11:53:43.208421+0000","last_clean":"2024-09-18T11:53:43.208421+0000","last_became_active":"2024-09-18T11:53:42.203696+0000","last_became_peered":"2024-09-18T11:53:42.203696+0000","last_unstale":"2024-09-18T11:53:45.234657+0000","last_undegraded":"2024-09-18T11:53:45.234657+0000","last_fullsized":"2024-09-18T11:53:45.234657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:50.811957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.225199+0000","last_change":"2024-09-18T11:53:42.212949+0000","last_active":"2024-09-18T11:53:44.225199+0000","last_peered":"2024-09-18T11:53:44.225199+0000","last_clean":"2024-09-18T11:53:44.225199+0000","last_became_active":"2024-09-18T11:53:42.211869+0000","last_became_peered":"2024-09-18T11:53:42.211869+0000","last_unstale":"2024-09-18T11:53:44.225199+0000","last_undegraded":"2024-09-18T11:53:44.225199+0000","last_fullsized":"2024-09-18T11:53:44.225199+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:34:23.038561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.227190+0000","last_change":"2024-09-18T11:53:42.209939+0000","last_active":"2024-09-18T11:53:44.227190+0000","last_peered":"2024-09-18T11:53:44.227190+0000","last_clean":"2024-09-18T11:53:44.227190+0000","last_became_active":"2024-09-18T11:53:42.209659+0000","last_became_peered":"2024-09-18T11:53:42.209659+0000","last_unstale":"2024-09-18T11:53:44.227190+0000","last_undegraded":"2024-09-18T11:53:44.227190+0000","last_fullsized":"2024-09-18T11:53:44.227190+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:24:24.722802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.233066+0000","last_change":"2024-09-18T11:53:45.233457+0000","last_active":"2024-09-18T11:53:44.225279+0000","last_peered":"2024-09-18T11:53:43.208142+0000","last_clean":"2024-09-18T11:53:43.208142+0000","last_became_active":"2024-09-18T11:53:42.203492+0000","last_became_peered":"2024-09-18T11:53:42.203492+0000","last_unstale":"2024-09-18T11:53:45.233066+0000","last_undegraded":"2024-09-18T11:53:45.233066+0000","last_fullsized":"2024-09-18T11:53:45.233066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:47:45.902672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.225007+0000","last_change":"2024-09-18T11:53:32.108045+0000","last_active":"2024-09-18T11:53:44.225007+0000","last_peered":"2024-09-18T11:53:44.225007+0000","last_clean":"2024-09-18T11:53:44.225007+0000","last_became_active":"2024-09-18T11:53:30.106253+0000","last_became_peered":"2024-09-18T11:53:30.106253+0000","last_unstale":"2024-09-18T11:53:44.225007+0000","last_undegraded":"2024-09-18T11:53:44.225007+0000","last_fullsized":"2024-09-18T11:53:44.225007+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:40:29.807545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.238702+0000","last_change":"2024-09-18T11:53:45.238922+0000","last_active":"2024-09-18T11:53:44.225949+0000","last_peered":"2024-09-18T11:53:43.212902+0000","last_clean":"2024-09-18T11:53:43.212902+0000","last_became_active":"2024-09-18T11:53:42.213577+0000","last_became_peered":"2024-09-18T11:53:42.213577+0000","last_unstale":"2024-09-18T11:53:45.238702+0000","last_undegraded":"2024-09-18T11:53:45.238702+0000","last_fullsized":"2024-09-18T11:53:45.238702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:55:55.551044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.233339+0000","last_change":"2024-09-18T11:53:45.233684+0000","last_active":"2024-09-18T11:53:29.091408+0000","last_peered":"2024-09-18T11:53:29.091408+0000","last_clean":"2024-09-18T11:53:29.091408+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.233339+0000","last_undegraded":"2024-09-18T11:53:45.233339+0000","last_fullsized":"2024-09-18T11:53:45.233339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:39:35.340135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.227034+0000","last_change":"2024-09-18T11:53:42.209716+0000","last_active":"2024-09-18T11:53:44.227034+0000","last_peered":"2024-09-18T11:53:44.227034+0000","last_clean":"2024-09-18T11:53:44.227034+0000","last_became_active":"2024-09-18T11:53:42.209437+0000","last_became_peered":"2024-09-18T11:53:42.209437+0000","last_unstale":"2024-09-18T11:53:44.227034+0000","last_undegraded":"2024-09-18T11:53:44.227034+0000","last_fullsized":"2024-09-18T11:53:44.227034+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:16:27.733240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":23,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.227203+0000","last_change":"2024-09-18T11:53:32.107187+0000","last_active":"2024-09-18T11:53:44.227203+0000","last_peered":"2024-09-18T11:53:44.227203+0000","last_clean":"2024-09-18T11:53:44.227203+0000","last_became_active":"2024-09-18T11:53:30.106303+0000","last_became_peered":"2024-09-18T11:53:30.106303+0000","last_unstale":"2024-09-18T11:53:44.227203+0000","last_undegraded":"2024-09-18T11:53:44.227203+0000","last_fullsized":"2024-09-18T11:53:44.227203+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:40:55.788481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.238217+0000","last_change":"2024-09-18T11:53:45.238639+0000","last_active":"2024-09-18T11:53:41.166841+0000","last_peered":"2024-09-18T11:53:41.166841+0000","last_clean":"2024-09-18T11:53:41.166841+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.238217+0000","last_undegraded":"2024-09-18T11:53:45.238217+0000","last_fullsized":"2024-09-18T11:53:45.238217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:01:45.711737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":26,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.239936+0000","last_change":"2024-09-18T11:53:45.240129+0000","last_active":"2024-09-18T11:53:44.226197+0000","last_peered":"2024-09-18T11:53:43.213133+0000","last_clean":"2024-09-18T11:53:43.213133+0000","last_became_active":"2024-09-18T11:53:30.112162+0000","last_became_peered":"2024-09-18T11:53:30.112162+0000","last_unstale":"2024-09-18T11:53:45.239936+0000","last_undegraded":"2024-09-18T11:53:45.239936+0000","last_fullsized":"2024-09-18T11:53:45.239936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:13:49.671400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.237648+0000","last_change":"2024-09-18T11:53:45.237972+0000","last_active":"2024-09-18T11:53:41.166841+0000","last_peered":"2024-09-18T11:53:41.166841+0000","last_clean":"2024-09-18T11:53:41.166841+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.237648+0000","last_undegraded":"2024-09-18T11:53:45.237648+0000","last_fullsized":"2024-09-18T11:53:45.237648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:38:09.530322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.227413+0000","last_change":"2024-09-18T11:53:42.213087+0000","last_active":"2024-09-18T11:53:44.227413+0000","last_peered":"2024-09-18T11:53:44.227413+0000","last_clean":"2024-09-18T11:53:44.227413+0000","last_became_active":"2024-09-18T11:53:42.211893+0000","last_became_peered":"2024-09-18T11:53:42.211893+0000","last_unstale":"2024-09-18T11:53:44.227413+0000","last_undegraded":"2024-09-18T11:53:44.227413+0000","last_fullsized":"2024-09-18T11:53:44.227413+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:46:15.719065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.225550+0000","last_change":"2024-09-18T11:53:32.108453+0000","last_active":"2024-09-18T11:53:44.225550+0000","last_peered":"2024-09-18T11:53:44.225550+0000","last_clean":"2024-09-18T11:53:44.225550+0000","last_became_active":"2024-09-18T11:53:30.106826+0000","last_became_peered":"2024-09-18T11:53:30.106826+0000","last_unstale":"2024-09-18T11:53:44.225550+0000","last_undegraded":"2024-09-18T11:53:44.225550+0000","last_fullsized":"2024-09-18T11:53:44.225550+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:46:21.716863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.226775+0000","last_change":"2024-09-18T11:53:42.210828+0000","last_active":"2024-09-18T11:53:44.226775+0000","last_peered":"2024-09-18T11:53:44.226775+0000","last_clean":"2024-09-18T11:53:44.226775+0000","last_became_active":"2024-09-18T11:53:42.210556+0000","last_became_peered":"2024-09-18T11:53:42.210556+0000","last_unstale":"2024-09-18T11:53:44.226775+0000","last_undegraded":"2024-09-18T11:53:44.226775+0000","last_fullsized":"2024-09-18T11:53:44.226775+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:39:13.072068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.235177+0000","last_change":"2024-09-18T11:53:45.235556+0000","last_active":"2024-09-18T11:53:29.091408+0000","last_peered":"2024-09-18T11:53:29.091408+0000","last_clean":"2024-09-18T11:53:29.091408+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.235177+0000","last_undegraded":"2024-09-18T11:53:45.235177+0000","last_fullsized":"2024-09-18T11:53:45.235177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:07:13.898948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.226611+0000","last_change":"2024-09-18T11:53:42.212946+0000","last_active":"2024-09-18T11:53:44.226611+0000","last_peered":"2024-09-18T11:53:44.226611+0000","last_clean":"2024-09-18T11:53:44.226611+0000","last_became_active":"2024-09-18T11:53:42.211801+0000","last_became_peered":"2024-09-18T11:53:42.211801+0000","last_unstale":"2024-09-18T11:53:44.226611+0000","last_undegraded":"2024-09-18T11:53:44.226611+0000","last_fullsized":"2024-09-18T11:53:44.226611+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:39:50.787931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.226920+0000","last_change":"2024-09-18T11:53:32.108476+0000","last_active":"2024-09-18T11:53:44.226920+0000","last_peered":"2024-09-18T11:53:44.226920+0000","last_clean":"2024-09-18T11:53:44.226920+0000","last_became_active":"2024-09-18T11:53:30.106924+0000","last_became_peered":"2024-09-18T11:53:30.106924+0000","last_unstale":"2024-09-18T11:53:44.226920+0000","last_undegraded":"2024-09-18T11:53:44.226920+0000","last_fullsized":"2024-09-18T11:53:44.226920+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:01.342017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":25,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.240035+0000","last_change":"2024-09-18T11:53:45.240199+0000","last_active":"2024-09-18T11:53:44.228107+0000","last_peered":"2024-09-18T11:53:43.213397+0000","last_clean":"2024-09-18T11:53:43.213397+0000","last_became_active":"2024-09-18T11:53:30.113507+0000","last_became_peered":"2024-09-18T11:53:30.113507+0000","last_unstale":"2024-09-18T11:53:45.240035+0000","last_undegraded":"2024-09-18T11:53:45.240035+0000","last_fullsized":"2024-09-18T11:53:45.240035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:50.421441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"18'1","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.238066+0000","last_change":"2024-09-18T11:53:42.209989+0000","last_active":"2024-09-18T11:53:45.238066+0000","last_peered":"2024-09-18T11:53:45.238066+0000","last_clean":"2024-09-18T11:53:45.238066+0000","last_became_active":"2024-09-18T11:53:42.209680+0000","last_became_peered":"2024-09-18T11:53:42.209680+0000","last_unstale":"2024-09-18T11:53:45.238066+0000","last_undegraded":"2024-09-18T11:53:45.238066+0000","last_fullsized":"2024-09-18T11:53:45.238066+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:52:12.355930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297520,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074320,"statfs":{"total":96636764160,"available":96332103680,"internally_reserved":0,"allocated":163840,"data_stored":42562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":151552,"data_stored":39182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:49.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:49.303 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:49.305 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:53:49.317 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-09-18T11:53:49.016242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":636,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595024,"kb_used_data":308,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148656,"statfs":{"total":193273528320,"available":192664223744,"internally_reserved":0,"allocated":315392,"data_stored":81744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.805400"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.230105+0000","last_change":"2024-09-18T11:53:42.210495+0000","last_active":"2024-09-18T11:53:44.230105+0000","last_peered":"2024-09-18T11:53:44.230105+0000","last_clean":"2024-09-18T11:53:44.230105+0000","last_became_active":"2024-09-18T11:53:42.210220+0000","last_became_peered":"2024-09-18T11:53:42.210220+0000","last_unstale":"2024-09-18T11:53:44.230105+0000","last_undegraded":"2024-09-18T11:53:44.230105+0000","last_fullsized":"2024-09-18T11:53:44.230105+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:15:30.898270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.235593+0000","last_change":"2024-09-18T11:53:45.235960+0000","last_active":"2024-09-18T11:53:41.166841+0000","last_peered":"2024-09-18T11:53:41.166841+0000","last_clean":"2024-09-18T11:53:41.166841+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.235593+0000","last_undegraded":"2024-09-18T11:53:45.235593+0000","last_fullsized":"2024-09-18T11:53:45.235593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:33:09.990682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.238821+0000","last_change":"2024-09-18T11:53:45.239008+0000","last_active":"2024-09-18T11:53:44.226410+0000","last_peered":"2024-09-18T11:53:43.212931+0000","last_clean":"2024-09-18T11:53:43.212931+0000","last_became_active":"2024-09-18T11:53:42.213226+0000","last_became_peered":"2024-09-18T11:53:42.213226+0000","last_unstale":"2024-09-18T11:53:45.238821+0000","last_undegraded":"2024-09-18T11:53:45.238821+0000","last_fullsized":"2024-09-18T11:53:45.238821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:10.557607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.233727+0000","last_change":"2024-09-18T11:53:45.234102+0000","last_active":"2024-09-18T11:53:41.166841+0000","last_peered":"2024-09-18T11:53:41.166841+0000","last_clean":"2024-09-18T11:53:41.166841+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.233727+0000","last_undegraded":"2024-09-18T11:53:45.233727+0000","last_fullsized":"2024-09-18T11:53:45.233727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:38:07.951416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.234657+0000","last_change":"2024-09-18T11:53:45.235027+0000","last_active":"2024-09-18T11:53:44.227135+0000","last_peered":"2024-09-18T11:53:43.208421+0000","last_clean":"2024-09-18T11:53:43.208421+0000","last_became_active":"2024-09-18T11:53:42.203696+0000","last_became_peered":"2024-09-18T11:53:42.203696+0000","last_unstale":"2024-09-18T11:53:45.234657+0000","last_undegraded":"2024-09-18T11:53:45.234657+0000","last_fullsized":"2024-09-18T11:53:45.234657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:50.811957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.225199+0000","last_change":"2024-09-18T11:53:42.212949+0000","last_active":"2024-09-18T11:53:44.225199+0000","last_peered":"2024-09-18T11:53:44.225199+0000","last_clean":"2024-09-18T11:53:44.225199+0000","last_became_active":"2024-09-18T11:53:42.211869+0000","last_became_peered":"2024-09-18T11:53:42.211869+0000","last_unstale":"2024-09-18T11:53:44.225199+0000","last_undegraded":"2024-09-18T11:53:44.225199+0000","last_fullsized":"2024-09-18T11:53:44.225199+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:34:23.038561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.227190+0000","last_change":"2024-09-18T11:53:42.209939+0000","last_active":"2024-09-18T11:53:44.227190+0000","last_peered":"2024-09-18T11:53:44.227190+0000","last_clean":"2024-09-18T11:53:44.227190+0000","last_became_active":"2024-09-18T11:53:42.209659+0000","last_became_peered":"2024-09-18T11:53:42.209659+0000","last_unstale":"2024-09-18T11:53:44.227190+0000","last_undegraded":"2024-09-18T11:53:44.227190+0000","last_fullsized":"2024-09-18T11:53:44.227190+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:24:24.722802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.233066+0000","last_change":"2024-09-18T11:53:45.233457+0000","last_active":"2024-09-18T11:53:44.225279+0000","last_peered":"2024-09-18T11:53:43.208142+0000","last_clean":"2024-09-18T11:53:43.208142+0000","last_became_active":"2024-09-18T11:53:42.203492+0000","last_became_peered":"2024-09-18T11:53:42.203492+0000","last_unstale":"2024-09-18T11:53:45.233066+0000","last_undegraded":"2024-09-18T11:53:45.233066+0000","last_fullsized":"2024-09-18T11:53:45.233066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:47:45.902672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.225007+0000","last_change":"2024-09-18T11:53:32.108045+0000","last_active":"2024-09-18T11:53:44.225007+0000","last_peered":"2024-09-18T11:53:44.225007+0000","last_clean":"2024-09-18T11:53:44.225007+0000","last_became_active":"2024-09-18T11:53:30.106253+0000","last_became_peered":"2024-09-18T11:53:30.106253+0000","last_unstale":"2024-09-18T11:53:44.225007+0000","last_undegraded":"2024-09-18T11:53:44.225007+0000","last_fullsized":"2024-09-18T11:53:44.225007+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:40:29.807545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.238702+0000","last_change":"2024-09-18T11:53:45.238922+0000","last_active":"2024-09-18T11:53:44.225949+0000","last_peered":"2024-09-18T11:53:43.212902+0000","last_clean":"2024-09-18T11:53:43.212902+0000","last_became_active":"2024-09-18T11:53:42.213577+0000","last_became_peered":"2024-09-18T11:53:42.213577+0000","last_unstale":"2024-09-18T11:53:45.238702+0000","last_undegraded":"2024-09-18T11:53:45.238702+0000","last_fullsized":"2024-09-18T11:53:45.238702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:55:55.551044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.233339+0000","last_change":"2024-09-18T11:53:45.233684+0000","last_active":"2024-09-18T11:53:29.091408+0000","last_peered":"2024-09-18T11:53:29.091408+0000","last_clean":"2024-09-18T11:53:29.091408+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.233339+0000","last_undegraded":"2024-09-18T11:53:45.233339+0000","last_fullsized":"2024-09-18T11:53:45.233339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:39:35.340135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.227034+0000","last_change":"2024-09-18T11:53:42.209716+0000","last_active":"2024-09-18T11:53:44.227034+0000","last_peered":"2024-09-18T11:53:44.227034+0000","last_clean":"2024-09-18T11:53:44.227034+0000","last_became_active":"2024-09-18T11:53:42.209437+0000","last_became_peered":"2024-09-18T11:53:42.209437+0000","last_unstale":"2024-09-18T11:53:44.227034+0000","last_undegraded":"2024-09-18T11:53:44.227034+0000","last_fullsized":"2024-09-18T11:53:44.227034+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:16:27.733240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":23,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.227203+0000","last_change":"2024-09-18T11:53:32.107187+0000","last_active":"2024-09-18T11:53:44.227203+0000","last_peered":"2024-09-18T11:53:44.227203+0000","last_clean":"2024-09-18T11:53:44.227203+0000","last_became_active":"2024-09-18T11:53:30.106303+0000","last_became_peered":"2024-09-18T11:53:30.106303+0000","last_unstale":"2024-09-18T11:53:44.227203+0000","last_undegraded":"2024-09-18T11:53:44.227203+0000","last_fullsized":"2024-09-18T11:53:44.227203+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:40:55.788481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.238217+0000","last_change":"2024-09-18T11:53:45.238639+0000","last_active":"2024-09-18T11:53:41.166841+0000","last_peered":"2024-09-18T11:53:41.166841+0000","last_clean":"2024-09-18T11:53:41.166841+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.238217+0000","last_undegraded":"2024-09-18T11:53:45.238217+0000","last_fullsized":"2024-09-18T11:53:45.238217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:01:45.711737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":26,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.239936+0000","last_change":"2024-09-18T11:53:45.240129+0000","last_active":"2024-09-18T11:53:44.226197+0000","last_peered":"2024-09-18T11:53:43.213133+0000","last_clean":"2024-09-18T11:53:43.213133+0000","last_became_active":"2024-09-18T11:53:30.112162+0000","last_became_peered":"2024-09-18T11:53:30.112162+0000","last_unstale":"2024-09-18T11:53:45.239936+0000","last_undegraded":"2024-09-18T11:53:45.239936+0000","last_fullsized":"2024-09-18T11:53:45.239936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:13:49.671400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.237648+0000","last_change":"2024-09-18T11:53:45.237972+0000","last_active":"2024-09-18T11:53:41.166841+0000","last_peered":"2024-09-18T11:53:41.166841+0000","last_clean":"2024-09-18T11:53:41.166841+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.237648+0000","last_undegraded":"2024-09-18T11:53:45.237648+0000","last_fullsized":"2024-09-18T11:53:45.237648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:38:09.530322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.227413+0000","last_change":"2024-09-18T11:53:42.213087+0000","last_active":"2024-09-18T11:53:44.227413+0000","last_peered":"2024-09-18T11:53:44.227413+0000","last_clean":"2024-09-18T11:53:44.227413+0000","last_became_active":"2024-09-18T11:53:42.211893+0000","last_became_peered":"2024-09-18T11:53:42.211893+0000","last_unstale":"2024-09-18T11:53:44.227413+0000","last_undegraded":"2024-09-18T11:53:44.227413+0000","last_fullsized":"2024-09-18T11:53:44.227413+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:46:15.719065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.225550+0000","last_change":"2024-09-18T11:53:32.108453+0000","last_active":"2024-09-18T11:53:44.225550+0000","last_peered":"2024-09-18T11:53:44.225550+0000","last_clean":"2024-09-18T11:53:44.225550+0000","last_became_active":"2024-09-18T11:53:30.106826+0000","last_became_peered":"2024-09-18T11:53:30.106826+0000","last_unstale":"2024-09-18T11:53:44.225550+0000","last_undegraded":"2024-09-18T11:53:44.225550+0000","last_fullsized":"2024-09-18T11:53:44.225550+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:46:21.716863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.226775+0000","last_change":"2024-09-18T11:53:42.210828+0000","last_active":"2024-09-18T11:53:44.226775+0000","last_peered":"2024-09-18T11:53:44.226775+0000","last_clean":"2024-09-18T11:53:44.226775+0000","last_became_active":"2024-09-18T11:53:42.210556+0000","last_became_peered":"2024-09-18T11:53:42.210556+0000","last_unstale":"2024-09-18T11:53:44.226775+0000","last_undegraded":"2024-09-18T11:53:44.226775+0000","last_fullsized":"2024-09-18T11:53:44.226775+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:39:13.072068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":7,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.235177+0000","last_change":"2024-09-18T11:53:45.235556+0000","last_active":"2024-09-18T11:53:29.091408+0000","last_peered":"2024-09-18T11:53:29.091408+0000","last_clean":"2024-09-18T11:53:29.091408+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:53:45.235177+0000","last_undegraded":"2024-09-18T11:53:45.235177+0000","last_fullsized":"2024-09-18T11:53:45.235177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:07:13.898948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.226611+0000","last_change":"2024-09-18T11:53:42.212946+0000","last_active":"2024-09-18T11:53:44.226611+0000","last_peered":"2024-09-18T11:53:44.226611+0000","last_clean":"2024-09-18T11:53:44.226611+0000","last_became_active":"2024-09-18T11:53:42.211801+0000","last_became_peered":"2024-09-18T11:53:42.211801+0000","last_unstale":"2024-09-18T11:53:44.226611+0000","last_undegraded":"2024-09-18T11:53:44.226611+0000","last_fullsized":"2024-09-18T11:53:44.226611+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:39:50.787931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T11:53:44.226920+0000","last_change":"2024-09-18T11:53:32.108476+0000","last_active":"2024-09-18T11:53:44.226920+0000","last_peered":"2024-09-18T11:53:44.226920+0000","last_clean":"2024-09-18T11:53:44.226920+0000","last_became_active":"2024-09-18T11:53:30.106924+0000","last_became_peered":"2024-09-18T11:53:30.106924+0000","last_unstale":"2024-09-18T11:53:44.226920+0000","last_undegraded":"2024-09-18T11:53:44.226920+0000","last_fullsized":"2024-09-18T11:53:44.226920+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:01.342017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":25,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-18T11:53:45.240035+0000","last_change":"2024-09-18T11:53:45.240199+0000","last_active":"2024-09-18T11:53:44.228107+0000","last_peered":"2024-09-18T11:53:43.213397+0000","last_clean":"2024-09-18T11:53:43.213397+0000","last_became_active":"2024-09-18T11:53:30.113507+0000","last_became_peered":"2024-09-18T11:53:30.113507+0000","last_unstale":"2024-09-18T11:53:45.240035+0000","last_undegraded":"2024-09-18T11:53:45.240035+0000","last_fullsized":"2024-09-18T11:53:45.240035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:50.421441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"18'1","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.238066+0000","last_change":"2024-09-18T11:53:42.209989+0000","last_active":"2024-09-18T11:53:45.238066+0000","last_peered":"2024-09-18T11:53:45.238066+0000","last_clean":"2024-09-18T11:53:45.238066+0000","last_became_active":"2024-09-18T11:53:42.209680+0000","last_became_peered":"2024-09-18T11:53:42.209680+0000","last_unstale":"2024-09-18T11:53:45.238066+0000","last_undegraded":"2024-09-18T11:53:45.238066+0000","last_fullsized":"2024-09-18T11:53:45.238066+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:52:12.355930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297520,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074320,"statfs":{"total":96636764160,"available":96332103680,"internally_reserved":0,"allocated":163840,"data_stored":42562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":151552,"data_stored":39182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:52.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:53:52.620 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:53:52.622 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:53:52.633 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-18T11:53:51.016587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":636,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595064,"kb_used_data":336,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148616,"statfs":{"total":193273528320,"available":192664182784,"internally_reserved":0,"allocated":344064,"data_stored":90534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.795471"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.234860+0000","last_change":"2024-09-18T11:53:42.210495+0000","last_active":"2024-09-18T11:53:45.234860+0000","last_peered":"2024-09-18T11:53:45.234860+0000","last_clean":"2024-09-18T11:53:45.234860+0000","last_became_active":"2024-09-18T11:53:42.210220+0000","last_became_peered":"2024-09-18T11:53:42.210220+0000","last_unstale":"2024-09-18T11:53:45.234860+0000","last_undegraded":"2024-09-18T11:53:45.234860+0000","last_fullsized":"2024-09-18T11:53:45.234860+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:15:30.898270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T11:53:46.247689+0000","last_change":"2024-09-18T11:53:46.247689+0000","last_active":"2024-09-18T11:53:46.247689+0000","last_peered":"2024-09-18T11:53:46.247689+0000","last_clean":"2024-09-18T11:53:46.247689+0000","last_became_active":"2024-09-18T11:53:46.247309+0000","last_became_peered":"2024-09-18T11:53:46.247309+0000","last_unstale":"2024-09-18T11:53:46.247689+0000","last_undegraded":"2024-09-18T11:53:46.247689+0000","last_fullsized":"2024-09-18T11:53:46.247689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:13:51.862381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":23,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T11:53:46.248844+0000","last_change":"2024-09-18T11:53:46.248844+0000","last_active":"2024-09-18T11:53:46.248844+0000","last_peered":"2024-09-18T11:53:46.248844+0000","last_clean":"2024-09-18T11:53:46.248844+0000","last_became_active":"2024-09-18T11:53:46.248497+0000","last_became_peered":"2024-09-18T11:53:46.248497+0000","last_unstale":"2024-09-18T11:53:46.248844+0000","last_undegraded":"2024-09-18T11:53:46.248844+0000","last_fullsized":"2024-09-18T11:53:46.248844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:57:56.248128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T11:53:46.245127+0000","last_change":"2024-09-18T11:53:46.245127+0000","last_active":"2024-09-18T11:53:46.245127+0000","last_peered":"2024-09-18T11:53:46.245127+0000","last_clean":"2024-09-18T11:53:46.245127+0000","last_became_active":"2024-09-18T11:53:46.244245+0000","last_became_peered":"2024-09-18T11:53:46.244245+0000","last_unstale":"2024-09-18T11:53:46.245127+0000","last_undegraded":"2024-09-18T11:53:46.245127+0000","last_fullsized":"2024-09-18T11:53:46.245127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:10:34.302691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":23,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T11:53:46.247503+0000","last_change":"2024-09-18T11:53:46.247503+0000","last_active":"2024-09-18T11:53:46.247503+0000","last_peered":"2024-09-18T11:53:46.247503+0000","last_clean":"2024-09-18T11:53:46.247503+0000","last_became_active":"2024-09-18T11:53:46.247134+0000","last_became_peered":"2024-09-18T11:53:46.247134+0000","last_unstale":"2024-09-18T11:53:46.247503+0000","last_undegraded":"2024-09-18T11:53:46.247503+0000","last_fullsized":"2024-09-18T11:53:46.247503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:34.459028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.237741+0000","last_change":"2024-09-18T11:53:42.212949+0000","last_active":"2024-09-18T11:53:45.237741+0000","last_peered":"2024-09-18T11:53:45.237741+0000","last_clean":"2024-09-18T11:53:45.237741+0000","last_became_active":"2024-09-18T11:53:42.211869+0000","last_became_peered":"2024-09-18T11:53:42.211869+0000","last_unstale":"2024-09-18T11:53:45.237741+0000","last_undegraded":"2024-09-18T11:53:45.237741+0000","last_fullsized":"2024-09-18T11:53:45.237741+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:34:23.038561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.238142+0000","last_change":"2024-09-18T11:53:42.209939+0000","last_active":"2024-09-18T11:53:45.238142+0000","last_peered":"2024-09-18T11:53:45.238142+0000","last_clean":"2024-09-18T11:53:45.238142+0000","last_became_active":"2024-09-18T11:53:42.209659+0000","last_became_peered":"2024-09-18T11:53:42.209659+0000","last_unstale":"2024-09-18T11:53:45.238142+0000","last_undegraded":"2024-09-18T11:53:45.238142+0000","last_fullsized":"2024-09-18T11:53:45.238142+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:24:24.722802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T11:53:46.245415+0000","last_change":"2024-09-18T11:53:46.245415+0000","last_active":"2024-09-18T11:53:46.245415+0000","last_peered":"2024-09-18T11:53:46.245415+0000","last_clean":"2024-09-18T11:53:46.245415+0000","last_became_active":"2024-09-18T11:53:46.245056+0000","last_became_peered":"2024-09-18T11:53:46.245056+0000","last_unstale":"2024-09-18T11:53:46.245415+0000","last_undegraded":"2024-09-18T11:53:46.245415+0000","last_fullsized":"2024-09-18T11:53:46.245415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:38:09.530322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.237084+0000","last_change":"2024-09-18T11:53:32.108045+0000","last_active":"2024-09-18T11:53:45.237084+0000","last_peered":"2024-09-18T11:53:45.237084+0000","last_clean":"2024-09-18T11:53:45.237084+0000","last_became_active":"2024-09-18T11:53:30.106253+0000","last_became_peered":"2024-09-18T11:53:30.106253+0000","last_unstale":"2024-09-18T11:53:45.237084+0000","last_undegraded":"2024-09-18T11:53:45.237084+0000","last_fullsized":"2024-09-18T11:53:45.237084+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:40:29.807545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T11:53:46.249252+0000","last_change":"2024-09-18T11:53:46.249252+0000","last_active":"2024-09-18T11:53:46.249252+0000","last_peered":"2024-09-18T11:53:46.249252+0000","last_clean":"2024-09-18T11:53:46.249252+0000","last_became_active":"2024-09-18T11:53:46.248622+0000","last_became_peered":"2024-09-18T11:53:46.248622+0000","last_unstale":"2024-09-18T11:53:46.249252+0000","last_undegraded":"2024-09-18T11:53:46.249252+0000","last_fullsized":"2024-09-18T11:53:46.249252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:07:10.256348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T11:53:46.245063+0000","last_change":"2024-09-18T11:53:46.245063+0000","last_active":"2024-09-18T11:53:46.245063+0000","last_peered":"2024-09-18T11:53:46.245063+0000","last_clean":"2024-09-18T11:53:46.245063+0000","last_became_active":"2024-09-18T11:53:46.244662+0000","last_became_peered":"2024-09-18T11:53:46.244662+0000","last_unstale":"2024-09-18T11:53:46.245063+0000","last_undegraded":"2024-09-18T11:53:46.245063+0000","last_fullsized":"2024-09-18T11:53:46.245063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:01:33.636304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.232857+0000","last_change":"2024-09-18T11:53:42.209716+0000","last_active":"2024-09-18T11:53:45.232857+0000","last_peered":"2024-09-18T11:53:45.232857+0000","last_clean":"2024-09-18T11:53:45.232857+0000","last_became_active":"2024-09-18T11:53:42.209437+0000","last_became_peered":"2024-09-18T11:53:42.209437+0000","last_unstale":"2024-09-18T11:53:45.232857+0000","last_undegraded":"2024-09-18T11:53:45.232857+0000","last_fullsized":"2024-09-18T11:53:45.232857+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:16:27.733240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.233041+0000","last_change":"2024-09-18T11:53:32.107187+0000","last_active":"2024-09-18T11:53:45.233041+0000","last_peered":"2024-09-18T11:53:45.233041+0000","last_clean":"2024-09-18T11:53:45.233041+0000","last_became_active":"2024-09-18T11:53:30.106303+0000","last_became_peered":"2024-09-18T11:53:30.106303+0000","last_unstale":"2024-09-18T11:53:45.233041+0000","last_undegraded":"2024-09-18T11:53:45.233041+0000","last_fullsized":"2024-09-18T11:53:45.233041+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:40:55.788481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T11:53:46.248805+0000","last_change":"2024-09-18T11:53:46.248805+0000","last_active":"2024-09-18T11:53:46.248805+0000","last_peered":"2024-09-18T11:53:46.248805+0000","last_clean":"2024-09-18T11:53:46.248805+0000","last_became_active":"2024-09-18T11:53:46.248466+0000","last_became_peered":"2024-09-18T11:53:46.248466+0000","last_unstale":"2024-09-18T11:53:46.248805+0000","last_undegraded":"2024-09-18T11:53:46.248805+0000","last_fullsized":"2024-09-18T11:53:46.248805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:18:24.717062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T11:53:46.255156+0000","last_change":"2024-09-18T11:53:46.255156+0000","last_active":"2024-09-18T11:53:46.255156+0000","last_peered":"2024-09-18T11:53:46.255156+0000","last_clean":"2024-09-18T11:53:46.255156+0000","last_became_active":"2024-09-18T11:53:46.249019+0000","last_became_peered":"2024-09-18T11:53:46.249019+0000","last_unstale":"2024-09-18T11:53:46.255156+0000","last_undegraded":"2024-09-18T11:53:46.255156+0000","last_fullsized":"2024-09-18T11:53:46.255156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:58:16.455955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T11:53:46.247985+0000","last_change":"2024-09-18T11:53:46.247985+0000","last_active":"2024-09-18T11:53:46.247985+0000","last_peered":"2024-09-18T11:53:46.247985+0000","last_clean":"2024-09-18T11:53:46.247985+0000","last_became_active":"2024-09-18T11:53:46.247635+0000","last_became_peered":"2024-09-18T11:53:46.247635+0000","last_unstale":"2024-09-18T11:53:46.247985+0000","last_undegraded":"2024-09-18T11:53:46.247985+0000","last_fullsized":"2024-09-18T11:53:46.247985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:59:41.680760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.239285+0000","last_change":"2024-09-18T11:53:42.213087+0000","last_active":"2024-09-18T11:53:45.239285+0000","last_peered":"2024-09-18T11:53:45.239285+0000","last_clean":"2024-09-18T11:53:45.239285+0000","last_became_active":"2024-09-18T11:53:42.211893+0000","last_became_peered":"2024-09-18T11:53:42.211893+0000","last_unstale":"2024-09-18T11:53:45.239285+0000","last_undegraded":"2024-09-18T11:53:45.239285+0000","last_fullsized":"2024-09-18T11:53:45.239285+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:46:15.719065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.238113+0000","last_change":"2024-09-18T11:53:32.108453+0000","last_active":"2024-09-18T11:53:45.238113+0000","last_peered":"2024-09-18T11:53:45.238113+0000","last_clean":"2024-09-18T11:53:45.238113+0000","last_became_active":"2024-09-18T11:53:30.106826+0000","last_became_peered":"2024-09-18T11:53:30.106826+0000","last_unstale":"2024-09-18T11:53:45.238113+0000","last_undegraded":"2024-09-18T11:53:45.238113+0000","last_fullsized":"2024-09-18T11:53:45.238113+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:46:21.716863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.239137+0000","last_change":"2024-09-18T11:53:42.210828+0000","last_active":"2024-09-18T11:53:45.239137+0000","last_peered":"2024-09-18T11:53:45.239137+0000","last_clean":"2024-09-18T11:53:45.239137+0000","last_became_active":"2024-09-18T11:53:42.210556+0000","last_became_peered":"2024-09-18T11:53:42.210556+0000","last_unstale":"2024-09-18T11:53:45.239137+0000","last_undegraded":"2024-09-18T11:53:45.239137+0000","last_fullsized":"2024-09-18T11:53:45.239137+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:39:13.072068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T11:53:46.247581+0000","last_change":"2024-09-18T11:53:46.247581+0000","last_active":"2024-09-18T11:53:46.247581+0000","last_peered":"2024-09-18T11:53:46.247581+0000","last_clean":"2024-09-18T11:53:46.247581+0000","last_became_active":"2024-09-18T11:53:46.247233+0000","last_became_peered":"2024-09-18T11:53:46.247233+0000","last_unstale":"2024-09-18T11:53:46.247581+0000","last_undegraded":"2024-09-18T11:53:46.247581+0000","last_fullsized":"2024-09-18T11:53:46.247581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:38:58.952412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.238482+0000","last_change":"2024-09-18T11:53:42.212946+0000","last_active":"2024-09-18T11:53:45.238482+0000","last_peered":"2024-09-18T11:53:45.238482+0000","last_clean":"2024-09-18T11:53:45.238482+0000","last_became_active":"2024-09-18T11:53:42.211801+0000","last_became_peered":"2024-09-18T11:53:42.211801+0000","last_unstale":"2024-09-18T11:53:45.238482+0000","last_undegraded":"2024-09-18T11:53:45.238482+0000","last_fullsized":"2024-09-18T11:53:45.238482+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:39:50.787931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.239241+0000","last_change":"2024-09-18T11:53:32.108476+0000","last_active":"2024-09-18T11:53:45.239241+0000","last_peered":"2024-09-18T11:53:45.239241+0000","last_clean":"2024-09-18T11:53:45.239241+0000","last_became_active":"2024-09-18T11:53:30.106924+0000","last_became_peered":"2024-09-18T11:53:30.106924+0000","last_unstale":"2024-09-18T11:53:45.239241+0000","last_undegraded":"2024-09-18T11:53:45.239241+0000","last_fullsized":"2024-09-18T11:53:45.239241+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:01.342017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-18T11:53:46.248234+0000","last_change":"2024-09-18T11:53:46.248234+0000","last_active":"2024-09-18T11:53:46.248234+0000","last_peered":"2024-09-18T11:53:46.248234+0000","last_clean":"2024-09-18T11:53:46.248234+0000","last_became_active":"2024-09-18T11:53:46.247904+0000","last_became_peered":"2024-09-18T11:53:46.247904+0000","last_unstale":"2024-09-18T11:53:46.248234+0000","last_undegraded":"2024-09-18T11:53:46.248234+0000","last_fullsized":"2024-09-18T11:53:46.248234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:08:15.913317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"18'1","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-18T11:53:45.296065+0000","last_change":"2024-09-18T11:53:42.209989+0000","last_active":"2024-09-18T11:53:45.296065+0000","last_peered":"2024-09-18T11:53:45.296065+0000","last_clean":"2024-09-18T11:53:45.296065+0000","last_became_active":"2024-09-18T11:53:42.209680+0000","last_became_peered":"2024-09-18T11:53:42.209680+0000","last_unstale":"2024-09-18T11:53:45.296065+0000","last_undegraded":"2024-09-18T11:53:45.296065+0000","last_fullsized":"2024-09-18T11:53:45.296065+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:52:12.355930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297532,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074308,"statfs":{"total":96636764160,"available":96332091392,"internally_reserved":0,"allocated":172032,"data_stored":45267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297532,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074308,"statfs":{"total":96636764160,"available":96332091392,"internally_reserved":0,"allocated":172032,"data_stored":45267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:53:52.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:recovered! 2024-09-18T11:53:52.634 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_1 /etc/fstab 2024-09-18T11:53:52.635 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_1 /etc/fstab 2024-09-18T11:53:52.685 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_1 /etc/fstab 2024-09-18T11:53:52.685 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_1 /etc/fstab 2024-09-18T11:53:52.733 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_1 2024-09-18T11:53:52.733 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_1 2024-09-18T11:53:52.779 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_2 /etc/fstab 2024-09-18T11:53:52.779 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_2 /etc/fstab 2024-09-18T11:53:52.826 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_2 /etc/fstab 2024-09-18T11:53:52.826 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_2 /etc/fstab 2024-09-18T11:53:52.874 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_2 2024-09-18T11:53:52.874 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_2 2024-09-18T11:53:52.920 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_3 /etc/fstab 2024-09-18T11:53:52.920 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_3 /etc/fstab 2024-09-18T11:53:52.967 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_3 /etc/fstab 2024-09-18T11:53:52.967 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_3 /etc/fstab 2024-09-18T11:53:53.018 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_3 2024-09-18T11:53:53.018 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_3 2024-09-18T11:53:53.064 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_4 /etc/fstab 2024-09-18T11:53:53.064 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_4 /etc/fstab 2024-09-18T11:53:53.114 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_4 /etc/fstab 2024-09-18T11:53:53.114 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_4 /etc/fstab 2024-09-18T11:53:53.158 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_4 2024-09-18T11:53:53.159 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_4 2024-09-18T11:53:53.209 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_5 /etc/fstab 2024-09-18T11:53:53.209 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_5 /etc/fstab 2024-09-18T11:53:53.256 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_5 /etc/fstab 2024-09-18T11:53:53.256 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_5 /etc/fstab 2024-09-18T11:53:53.304 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_5 2024-09-18T11:53:53.304 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_5 2024-09-18T11:53:53.350 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_6 /etc/fstab 2024-09-18T11:53:53.351 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_6 /etc/fstab 2024-09-18T11:53:53.397 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_6 /etc/fstab 2024-09-18T11:53:53.397 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_6 /etc/fstab 2024-09-18T11:53:53.447 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_6 2024-09-18T11:53:53.447 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_6 2024-09-18T11:53:53.492 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_7 /etc/fstab 2024-09-18T11:53:53.493 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_7 /etc/fstab 2024-09-18T11:53:53.588 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_7 /etc/fstab 2024-09-18T11:53:53.588 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_7 /etc/fstab 2024-09-18T11:53:53.635 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_7 2024-09-18T11:53:53.636 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_7 2024-09-18T11:53:53.682 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_8 /etc/fstab 2024-09-18T11:53:53.683 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_8 /etc/fstab 2024-09-18T11:53:53.731 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_8 /etc/fstab 2024-09-18T11:53:53.732 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_8 /etc/fstab 2024-09-18T11:53:53.781 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_8 2024-09-18T11:53:53.781 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_8 2024-09-18T11:53:53.827 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_9 /etc/fstab 2024-09-18T11:53:53.827 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_9 /etc/fstab 2024-09-18T11:53:53.873 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_9 /etc/fstab 2024-09-18T11:53:53.873 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_9 /etc/fstab 2024-09-18T11:53:53.917 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_9 2024-09-18T11:53:53.917 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_9 2024-09-18T11:53:53.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 injectargs '--osd-recovery-delay-start 1000 --osd-min-pg-log-entries 100000000' 2024-09-18T11:53:54.103 INFO:teuthology.orchestra.run.smithi154.stdout:{} 2024-09-18T11:53:54.105 INFO:teuthology.orchestra.run.smithi154.stderr:osd_min_pg_log_entries = '' (not observed, change may require restart) osd_recovery_delay_start = '' 2024-09-18T11:53:54.116 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-18T11:53:54.116 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:53:54.179 INFO:tasks.ceph.osd.0:Stopped 2024-09-18T11:53:54.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2024-09-18T11:53:54.463 INFO:tasks.ceph.mon.a.smithi154.stderr:2024-09-18T11:53:54.457+0000 7f2c7dcba700 -1 mon.a@0(leader).osd e20 definitely_dead 0 2024-09-18T11:53:54.467 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0 is already down. 2024-09-18T11:53:54.477 INFO:tasks.util.rados:rados -p unfounddel_pool put new_1 /etc/fstab 2024-09-18T11:53:54.477 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_1 /etc/fstab 2024-09-18T11:53:55.291 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_1 /etc/fstab 2024-09-18T11:53:55.291 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_1 /etc/fstab 2024-09-18T11:53:55.334 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_1 /etc/fstab 2024-09-18T11:53:55.335 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_1 /etc/fstab 2024-09-18T11:53:55.424 INFO:tasks.util.rados:rados -p unfounddel_pool put new_2 /etc/fstab 2024-09-18T11:53:55.425 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_2 /etc/fstab 2024-09-18T11:53:55.470 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_2 /etc/fstab 2024-09-18T11:53:55.471 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_2 /etc/fstab 2024-09-18T11:53:55.519 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_2 /etc/fstab 2024-09-18T11:53:55.519 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_2 /etc/fstab 2024-09-18T11:53:55.567 INFO:tasks.util.rados:rados -p unfounddel_pool put new_3 /etc/fstab 2024-09-18T11:53:55.567 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_3 /etc/fstab 2024-09-18T11:53:55.618 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_3 /etc/fstab 2024-09-18T11:53:55.619 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_3 /etc/fstab 2024-09-18T11:53:55.662 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_3 /etc/fstab 2024-09-18T11:53:55.662 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_3 /etc/fstab 2024-09-18T11:53:55.712 INFO:tasks.util.rados:rados -p unfounddel_pool put new_4 /etc/fstab 2024-09-18T11:53:55.713 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_4 /etc/fstab 2024-09-18T11:53:55.761 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_4 /etc/fstab 2024-09-18T11:53:55.761 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_4 /etc/fstab 2024-09-18T11:53:55.813 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_4 /etc/fstab 2024-09-18T11:53:55.813 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_4 /etc/fstab 2024-09-18T11:53:55.864 INFO:tasks.util.rados:rados -p unfounddel_pool put new_5 /etc/fstab 2024-09-18T11:53:55.864 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_5 /etc/fstab 2024-09-18T11:53:55.916 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_5 /etc/fstab 2024-09-18T11:53:55.916 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_5 /etc/fstab 2024-09-18T11:53:55.966 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_5 /etc/fstab 2024-09-18T11:53:55.966 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_5 /etc/fstab 2024-09-18T11:53:56.013 INFO:tasks.util.rados:rados -p unfounddel_pool put new_6 /etc/fstab 2024-09-18T11:53:56.013 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_6 /etc/fstab 2024-09-18T11:53:56.063 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_6 /etc/fstab 2024-09-18T11:53:56.063 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_6 /etc/fstab 2024-09-18T11:53:56.108 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_6 /etc/fstab 2024-09-18T11:53:56.108 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_6 /etc/fstab 2024-09-18T11:53:56.156 INFO:tasks.util.rados:rados -p unfounddel_pool put new_7 /etc/fstab 2024-09-18T11:53:56.157 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_7 /etc/fstab 2024-09-18T11:53:56.206 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_7 /etc/fstab 2024-09-18T11:53:56.206 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_7 /etc/fstab 2024-09-18T11:53:56.254 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_7 /etc/fstab 2024-09-18T11:53:56.254 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_7 /etc/fstab 2024-09-18T11:53:56.301 INFO:tasks.util.rados:rados -p unfounddel_pool put new_8 /etc/fstab 2024-09-18T11:53:56.301 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_8 /etc/fstab 2024-09-18T11:53:56.347 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_8 /etc/fstab 2024-09-18T11:53:56.347 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_8 /etc/fstab 2024-09-18T11:53:56.398 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_8 /etc/fstab 2024-09-18T11:53:56.399 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_8 /etc/fstab 2024-09-18T11:53:56.449 INFO:tasks.util.rados:rados -p unfounddel_pool put new_9 /etc/fstab 2024-09-18T11:53:56.449 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_9 /etc/fstab 2024-09-18T11:53:56.499 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_9 /etc/fstab 2024-09-18T11:53:56.499 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_9 /etc/fstab 2024-09-18T11:53:56.549 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_9 /etc/fstab 2024-09-18T11:53:56.550 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_9 /etc/fstab 2024-09-18T11:53:56.602 INFO:tasks.rep_lost_unfound_delete:osd.0 command_args is foo 2024-09-18T11:53:56.603 INFO:tasks.rep_lost_unfound_delete:() 2024-09-18T11:53:56.603 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-18T11:53:56.603 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 --osd-recovery-delay-start 1000 2024-09-18T11:53:56.605 INFO:tasks.ceph.osd.0:Started 2024-09-18T11:53:56.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T11:53:56.713 INFO:teuthology.orchestra.run.smithi154.stderr:no valid command found; 10 closest matches: 2024-09-18T11:53:56.713 INFO:teuthology.orchestra.run.smithi154.stderr:0 2024-09-18T11:53:56.713 INFO:teuthology.orchestra.run.smithi154.stderr:1 2024-09-18T11:53:56.713 INFO:teuthology.orchestra.run.smithi154.stderr:2 2024-09-18T11:53:56.713 INFO:teuthology.orchestra.run.smithi154.stderr:abort 2024-09-18T11:53:56.714 INFO:teuthology.orchestra.run.smithi154.stderr:assert 2024-09-18T11:53:56.714 INFO:teuthology.orchestra.run.smithi154.stderr:config diff 2024-09-18T11:53:56.714 INFO:teuthology.orchestra.run.smithi154.stderr:config diff get 2024-09-18T11:53:56.714 INFO:teuthology.orchestra.run.smithi154.stderr:config get 2024-09-18T11:53:56.714 INFO:teuthology.orchestra.run.smithi154.stderr:config help [] 2024-09-18T11:53:56.714 INFO:teuthology.orchestra.run.smithi154.stderr:config set ... 2024-09-18T11:53:56.714 INFO:teuthology.orchestra.run.smithi154.stderr:admin_socket: invalid command 2024-09-18T11:53:56.714 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T11:53:56.715 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-18T11:53:57.230 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:53:57.225+0000 7f6200387540 -1 Falling back to public interface 2024-09-18T11:53:59.815 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:53:59.809+0000 7f6200387540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T11:54:00.596 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:00.589+0000 7f6200387540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T11:54:00.939 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:00.933+0000 7f6200387540 -1 osd.0 19 log_to_monitors true 2024-09-18T11:54:01.303 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:01.297+0000 7f61f60cd700 -1 osd.0 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:54:01.716 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-18T11:54:01.826 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:01.827 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-18T11:54:01.827 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-18T11:54:01.827 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:01.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-09-18T11:54:02.130 INFO:tasks.ceph.mon.a.smithi154.stderr:2024-09-18T11:54:02.125+0000 7f2c7dcba700 -1 mon.a@0(leader).osd e22 definitely_dead 0 2024-09-18T11:54:02.134 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0 is already in. 2024-09-18T11:54:02.145 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for osd.0 to be up 2024-09-18T11:54:02.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:54:02.431 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:02.431 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":23,"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","created":"2024-09-18T11:53:20.028520+0000","modified":"2024-09-18T11:54:02.290734+0000","last_up_change":"2024-09-18T11:54:02.290734+0000","last_in_change":"2024-09-18T11:53:45.217859+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:53:29.033564+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-09-18T11:53:40.807258+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":19,"up_from":23,"up_thru":18,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":26420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":26420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":26420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":26420}]},"public_addr":"172.21.15.154:6800/26420","cluster_addr":"172.21.15.154:6801/26420","heartbeat_back_addr":"172.21.15.154:6803/26420","heartbeat_front_addr":"172.21.15.154:6802/26420","state":["exists","up"]},{"osd":1,"uuid":"fd6ca4f9-93c5-4509-8fea-0121afa900f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":21134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":21134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":21134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":21134}]},"public_addr":"172.21.15.154:6804/21134","cluster_addr":"172.21.15.154:6805/21134","heartbeat_back_addr":"172.21.15.154:6807/21134","heartbeat_front_addr":"172.21.15.154:6806/21134","state":["exists","up"]},{"osd":2,"uuid":"1554e072-019d-4d8e-b2ee-c0aac5a5e811","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":21136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":21136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":21136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":21136}]},"public_addr":"172.21.15.154:6800/21136","cluster_addr":"172.21.15.154:6801/21136","heartbeat_back_addr":"172.21.15.154:6803/21136","heartbeat_front_addr":"172.21.15.154:6802/21136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.685684+0000","dead_epoch":20},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.545391+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-18T11:53:26.679601+0000","dead_epoch":17}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:54:02.450 INFO:tasks.rep_lost_unfound_delete.ceph_manager:osd.0 is up 2024-09-18T11:54:02.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:54:02.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T11:54:02.594 INFO:teuthology.orchestra.run.smithi154.stdout:98784247810 2024-09-18T11:54:02.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:54:02.596 INFO:teuthology.orchestra.run.smithi154.stdout:34359738381 2024-09-18T11:54:02.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:54:02.865 INFO:teuthology.orchestra.run.smithi154.stdout:34359738380 2024-09-18T11:54:02.872 INFO:teuthology.orchestra.run.smithi154.stdout:34359738378 2024-09-18T11:54:02.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738381 got 34359738380 for osd.1 2024-09-18T11:54:02.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 98784247810 got 34359738378 for osd.0 2024-09-18T11:54:03.880 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T11:54:03.891 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:54:04.189 INFO:teuthology.orchestra.run.smithi154.stdout:34359738381 2024-09-18T11:54:04.202 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738381 got 34359738381 for osd.1 2024-09-18T11:54:04.202 DEBUG:teuthology.parallel:result is None 2024-09-18T11:54:04.222 INFO:teuthology.orchestra.run.smithi154.stdout:98784247810 2024-09-18T11:54:04.236 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 98784247810 got 98784247810 for osd.0 2024-09-18T11:54:04.236 DEBUG:teuthology.parallel:result is None 2024-09-18T11:54:04.236 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting till active 2024-09-18T11:54:04.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:04.519 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:04.522 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:54:04.532 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-18T11:54:03.295920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":14827,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51,"ondisk_log_size":51,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595688,"kb_used_data":528,"kb_used_omap":6,"kb_used_meta":595129,"kb_avail":188147992,"statfs":{"total":193273528320,"available":192663543808,"internally_reserved":0,"allocated":540672,"data_stored":131306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6306,"internal_metadata":609412958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8638,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.277840"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.466374+0000","last_change":"2024-09-18T11:54:02.298759+0000","last_active":"2024-09-18T11:54:01.300179+0000","last_peered":"2024-09-18T11:53:42.210495+0000","last_clean":"2024-09-18T11:53:42.210495+0000","last_became_active":"2024-09-18T11:53:42.210220+0000","last_became_peered":"2024-09-18T11:53:42.210220+0000","last_unstale":"2024-09-18T11:54:02.466374+0000","last_undegraded":"2024-09-18T11:54:02.466374+0000","last_fullsized":"2024-09-18T11:54:02.466374+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:05.481799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":21,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.466539+0000","last_change":"2024-09-18T11:54:02.298625+0000","last_active":"2024-09-18T11:54:02.298406+0000","last_peered":"2024-09-18T11:53:53.108530+0000","last_clean":"2024-09-18T11:53:53.108530+0000","last_became_active":"2024-09-18T11:53:46.247309+0000","last_became_peered":"2024-09-18T11:53:46.247309+0000","last_unstale":"2024-09-18T11:54:02.466539+0000","last_undegraded":"2024-09-18T11:54:02.466539+0000","last_fullsized":"2024-09-18T11:54:02.466539+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:04:36.987274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":39,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.461838+0000","last_change":"2024-09-18T11:54:02.461838+0000","last_active":"2024-09-18T11:54:02.461436+0000","last_peered":"2024-09-18T11:54:01.295683+0000","last_clean":"2024-09-18T11:53:53.677796+0000","last_became_active":"2024-09-18T11:53:55.281216+0000","last_became_peered":"2024-09-18T11:53:55.281216+0000","last_unstale":"2024-09-18T11:54:02.461838+0000","last_undegraded":"2024-09-18T11:54:02.461838+0000","last_fullsized":"2024-09-18T11:54:02.461838+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:00:42.890017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.465970+0000","last_change":"2024-09-18T11:54:02.300738+0000","last_active":"2024-09-18T11:54:02.300534+0000","last_peered":"2024-09-18T11:53:46.245127+0000","last_clean":"2024-09-18T11:53:46.245127+0000","last_became_active":"2024-09-18T11:53:46.244245+0000","last_became_peered":"2024-09-18T11:53:46.244245+0000","last_unstale":"2024-09-18T11:54:02.465970+0000","last_undegraded":"2024-09-18T11:54:02.465970+0000","last_fullsized":"2024-09-18T11:54:02.465970+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:57:37.436855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"19'3","reported_seq":30,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.466664+0000","last_change":"2024-09-18T11:54:02.297233+0000","last_active":"2024-09-18T11:54:02.297013+0000","last_peered":"2024-09-18T11:53:53.013443+0000","last_clean":"2024-09-18T11:53:53.013443+0000","last_became_active":"2024-09-18T11:53:46.247134+0000","last_became_peered":"2024-09-18T11:53:46.247134+0000","last_unstale":"2024-09-18T11:54:02.466664+0000","last_undegraded":"2024-09-18T11:54:02.466664+0000","last_fullsized":"2024-09-18T11:54:02.466664+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:52:44.941625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":28,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.463019+0000","last_change":"2024-09-18T11:54:02.463019+0000","last_active":"2024-09-18T11:54:02.462538+0000","last_peered":"2024-09-18T11:54:01.295225+0000","last_clean":"2024-09-18T11:53:53.869066+0000","last_became_active":"2024-09-18T11:53:55.278624+0000","last_became_peered":"2024-09-18T11:53:55.278624+0000","last_unstale":"2024-09-18T11:54:02.463019+0000","last_undegraded":"2024-09-18T11:54:02.463019+0000","last_fullsized":"2024-09-18T11:54:02.463019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:10:29.718476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"21'4","reported_seq":29,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.456740+0000","last_change":"2024-09-18T11:54:02.456740+0000","last_active":"2024-09-18T11:54:02.456308+0000","last_peered":"2024-09-18T11:54:01.295555+0000","last_clean":"2024-09-18T11:53:53.251624+0000","last_became_active":"2024-09-18T11:53:55.278338+0000","last_became_peered":"2024-09-18T11:53:55.278338+0000","last_unstale":"2024-09-18T11:54:02.456740+0000","last_undegraded":"2024-09-18T11:54:02.456740+0000","last_fullsized":"2024-09-18T11:54:02.456740+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:10:50.142128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.462835+0000","last_change":"2024-09-18T11:54:02.296677+0000","last_active":"2024-09-18T11:54:02.296427+0000","last_peered":"2024-09-18T11:53:46.245415+0000","last_clean":"2024-09-18T11:53:46.245415+0000","last_became_active":"2024-09-18T11:53:46.245056+0000","last_became_peered":"2024-09-18T11:53:46.245056+0000","last_unstale":"2024-09-18T11:54:02.462835+0000","last_undegraded":"2024-09-18T11:54:02.462835+0000","last_fullsized":"2024-09-18T11:54:02.462835+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:09:48.291198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":34,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.460164+0000","last_change":"2024-09-18T11:54:02.460573+0000","last_active":"2024-09-18T11:54:02.456501+0000","last_peered":"2024-09-18T11:54:01.295094+0000","last_clean":"2024-09-18T11:53:46.242383+0000","last_became_active":"2024-09-18T11:53:55.278478+0000","last_became_peered":"2024-09-18T11:53:55.278478+0000","last_unstale":"2024-09-18T11:54:02.460164+0000","last_undegraded":"2024-09-18T11:54:02.460164+0000","last_fullsized":"2024-09-18T11:54:02.460164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:10:40.632283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'5","reported_seq":41,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.462603+0000","last_change":"2024-09-18T11:54:02.462603+0000","last_active":"2024-09-18T11:54:02.462246+0000","last_peered":"2024-09-18T11:54:01.295327+0000","last_clean":"2024-09-18T11:53:53.726368+0000","last_became_active":"2024-09-18T11:53:55.278351+0000","last_became_peered":"2024-09-18T11:53:55.278351+0000","last_unstale":"2024-09-18T11:54:02.462603+0000","last_undegraded":"2024-09-18T11:54:02.462603+0000","last_fullsized":"2024-09-18T11:54:02.462603+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:26:22.553547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.467623+0000","last_change":"2024-09-18T11:54:02.296396+0000","last_active":"2024-09-18T11:54:02.296185+0000","last_peered":"2024-09-18T11:53:46.245063+0000","last_clean":"2024-09-18T11:53:46.245063+0000","last_became_active":"2024-09-18T11:53:46.244662+0000","last_became_peered":"2024-09-18T11:53:46.244662+0000","last_unstale":"2024-09-18T11:54:02.467623+0000","last_undegraded":"2024-09-18T11:54:02.467623+0000","last_fullsized":"2024-09-18T11:54:02.467623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:08.311459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"19'4","reported_seq":20,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.462817+0000","last_change":"2024-09-18T11:54:02.300981+0000","last_active":"2024-09-18T11:54:02.300734+0000","last_peered":"2024-09-18T11:53:53.913371+0000","last_clean":"2024-09-18T11:53:53.913371+0000","last_became_active":"2024-09-18T11:53:42.209437+0000","last_became_peered":"2024-09-18T11:53:42.209437+0000","last_unstale":"2024-09-18T11:54:02.462817+0000","last_undegraded":"2024-09-18T11:54:02.462817+0000","last_fullsized":"2024-09-18T11:54:02.462817+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:22:39.861059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":24,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.462295+0000","last_change":"2024-09-18T11:54:02.298302+0000","last_active":"2024-09-18T11:54:01.300454+0000","last_peered":"2024-09-18T11:53:39.957002+0000","last_clean":"2024-09-18T11:53:39.957002+0000","last_became_active":"2024-09-18T11:53:30.106303+0000","last_became_peered":"2024-09-18T11:53:30.106303+0000","last_unstale":"2024-09-18T11:54:02.462295+0000","last_undegraded":"2024-09-18T11:54:02.462295+0000","last_fullsized":"2024-09-18T11:54:02.462295+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:04:32.999446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'3","reported_seq":31,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.458445+0000","last_change":"2024-09-18T11:54:02.458445+0000","last_active":"2024-09-18T11:54:02.458043+0000","last_peered":"2024-09-18T11:54:01.295256+0000","last_clean":"2024-09-18T11:53:52.962496+0000","last_became_active":"2024-09-18T11:53:55.277934+0000","last_became_peered":"2024-09-18T11:53:55.277934+0000","last_unstale":"2024-09-18T11:54:02.458445+0000","last_undegraded":"2024-09-18T11:54:02.458445+0000","last_fullsized":"2024-09-18T11:54:02.458445+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:33:56.738375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":50,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.459074+0000","last_change":"2024-09-18T11:54:02.459750+0000","last_active":"2024-09-18T11:54:02.458993+0000","last_peered":"2024-09-18T11:54:01.295650+0000","last_clean":"2024-09-18T11:53:46.255648+0000","last_became_active":"2024-09-18T11:53:55.278783+0000","last_became_peered":"2024-09-18T11:53:55.278783+0000","last_unstale":"2024-09-18T11:54:02.459074+0000","last_undegraded":"2024-09-18T11:54:02.459074+0000","last_fullsized":"2024-09-18T11:54:02.459074+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:52:33.418981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":32,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.456549+0000","last_change":"2024-09-18T11:54:02.456549+0000","last_active":"2024-09-18T11:54:02.456186+0000","last_peered":"2024-09-18T11:54:01.294863+0000","last_clean":"2024-09-18T11:53:53.346427+0000","last_became_active":"2024-09-18T11:53:55.278007+0000","last_became_peered":"2024-09-18T11:53:55.278007+0000","last_unstale":"2024-09-18T11:54:02.456549+0000","last_undegraded":"2024-09-18T11:54:02.456549+0000","last_fullsized":"2024-09-18T11:54:02.456549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:58:49.486547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"21'1","reported_seq":25,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.463167+0000","last_change":"2024-09-18T11:54:02.463471+0000","last_active":"2024-09-18T11:54:02.463130+0000","last_peered":"2024-09-18T11:54:01.295979+0000","last_clean":"2024-09-18T11:53:46.243258+0000","last_became_active":"2024-09-18T11:53:55.279015+0000","last_became_peered":"2024-09-18T11:53:55.279015+0000","last_unstale":"2024-09-18T11:54:02.463167+0000","last_undegraded":"2024-09-18T11:54:02.463167+0000","last_fullsized":"2024-09-18T11:54:02.463167+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:17:45.170649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.457541+0000","last_change":"2024-09-18T11:54:02.458001+0000","last_active":"2024-09-18T11:54:02.457478+0000","last_peered":"2024-09-18T11:54:01.296005+0000","last_clean":"2024-09-18T11:53:46.242675+0000","last_became_active":"2024-09-18T11:53:55.278382+0000","last_became_peered":"2024-09-18T11:53:55.278382+0000","last_unstale":"2024-09-18T11:54:02.457541+0000","last_undegraded":"2024-09-18T11:54:02.457541+0000","last_fullsized":"2024-09-18T11:54:02.457541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:37:36.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'5","reported_seq":30,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.465269+0000","last_change":"2024-09-18T11:54:02.465269+0000","last_active":"2024-09-18T11:54:02.465008+0000","last_peered":"2024-09-18T11:54:01.295890+0000","last_clean":"2024-09-18T11:53:53.823053+0000","last_became_active":"2024-09-18T11:53:55.286944+0000","last_became_peered":"2024-09-18T11:53:55.286944+0000","last_unstale":"2024-09-18T11:54:02.465269+0000","last_undegraded":"2024-09-18T11:54:02.465269+0000","last_fullsized":"2024-09-18T11:54:02.465269+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:05:48.525151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.465380+0000","last_change":"2024-09-18T11:54:02.298454+0000","last_active":"2024-09-18T11:54:02.298206+0000","last_peered":"2024-09-18T11:53:46.247581+0000","last_clean":"2024-09-18T11:53:46.247581+0000","last_became_active":"2024-09-18T11:53:46.247233+0000","last_became_peered":"2024-09-18T11:53:46.247233+0000","last_unstale":"2024-09-18T11:54:02.465380+0000","last_undegraded":"2024-09-18T11:54:02.465380+0000","last_fullsized":"2024-09-18T11:54:02.465380+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:01:49.096451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"21'2","reported_seq":26,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.463488+0000","last_change":"2024-09-18T11:54:02.463718+0000","last_active":"2024-09-18T11:54:02.463459+0000","last_peered":"2024-09-18T11:54:01.295826+0000","last_clean":"2024-09-18T11:53:46.242858+0000","last_became_active":"2024-09-18T11:53:55.281799+0000","last_became_peered":"2024-09-18T11:53:55.281799+0000","last_unstale":"2024-09-18T11:54:02.463488+0000","last_undegraded":"2024-09-18T11:54:02.463488+0000","last_fullsized":"2024-09-18T11:54:02.463488+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:42:11.470641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.458666+0000","last_change":"2024-09-18T11:54:02.458955+0000","last_active":"2024-09-18T11:54:02.458629+0000","last_peered":"2024-09-18T11:54:01.295942+0000","last_clean":"2024-09-18T11:53:46.243252+0000","last_became_active":"2024-09-18T11:53:55.279582+0000","last_became_peered":"2024-09-18T11:53:55.279582+0000","last_unstale":"2024-09-18T11:54:02.458666+0000","last_undegraded":"2024-09-18T11:54:02.458666+0000","last_fullsized":"2024-09-18T11:54:02.458666+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:28:34.444869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":45,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.464346+0000","last_change":"2024-09-18T11:54:02.464513+0000","last_active":"2024-09-18T11:54:02.464318+0000","last_peered":"2024-09-18T11:54:01.296158+0000","last_clean":"2024-09-18T11:53:46.252690+0000","last_became_active":"2024-09-18T11:53:55.279204+0000","last_became_peered":"2024-09-18T11:53:55.279204+0000","last_unstale":"2024-09-18T11:54:02.464346+0000","last_undegraded":"2024-09-18T11:54:02.464346+0000","last_fullsized":"2024-09-18T11:54:02.464346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:50:59.872188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"21'9","reported_seq":34,"reported_epoch":23,"state":"peering","last_fresh":"2024-09-18T11:54:02.460273+0000","last_change":"2024-09-18T11:54:02.460273+0000","last_active":"2024-09-18T11:54:02.459722+0000","last_peered":"2024-09-18T11:54:01.295629+0000","last_clean":"2024-09-18T11:53:53.582527+0000","last_became_active":"2024-09-18T11:53:55.278118+0000","last_became_peered":"2024-09-18T11:53:55.278118+0000","last_unstale":"2024-09-18T11:54:02.460273+0000","last_undegraded":"2024-09-18T11:54:02.460273+0000","last_fullsized":"2024-09-18T11:54:02.460273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:28:28.605255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":14808,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":23446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48,"ondisk_log_size":48,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738381,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297656,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074184,"statfs":{"total":96636764160,"available":96331964416,"internally_reserved":0,"allocated":307200,"data_stored":71206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247810,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298032,"kb_used_data":228,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94073808,"statfs":{"total":96636764160,"available":96331579392,"internally_reserved":0,"allocated":233472,"data_stored":60100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:04.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:04.869 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:04.869 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":44,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":24,"num_osds":3,"num_up_osds":2,"osd_up_since":1726660442,"num_in_osds":2,"osd_in_since":1726660425,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":24}],"num_pgs":24,"num_pools":2,"num_objects":26,"data_bytes":14827,"bytes_used":609984512,"bytes_avail":192663543808,"bytes_total":193273528320,"inactive_pgs_ratio":1,"write_bytes_sec":2651,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-18T11:53:47.015854+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:54:04.883 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': 'c6912087-b15b-4223-a3df-3a06756ee775', 'health': {'status': 'HEALTH_OK', 'checks': {}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 44, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 24, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1726660442, 'num_in_osds': 2, 'osd_in_since': 1726660425, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'peering', 'count': 24}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 26, 'data_bytes': 14827, 'bytes_used': 609984512, 'bytes_avail': 192663543808, 'bytes_total': 193273528320, 'inactive_pgs_ratio': 1, 'write_bytes_sec': 2651, 'read_op_per_sec': 0, 'write_op_per_sec': 2}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 2, 'modified': '2024-09-18T11:53:47.015854+0000', 'services': {'mgr': {'daemons': {'summary': '', 'x': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'mon': {'daemons': {'summary': '', 'a': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'b': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'c': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '1': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-18T11:54:07.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:08.188 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:08.191 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:54:08.201 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-09-18T11:54:07.021484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17295,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595732,"kb_used_data":560,"kb_used_omap":6,"kb_used_meta":595129,"kb_avail":188147948,"statfs":{"total":193273528320,"available":192663498752,"internally_reserved":0,"allocated":573440,"data_stored":143578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6306,"internal_metadata":609412958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":-3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002270"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.313823+0000","last_change":"2024-09-18T11:54:03.313823+0000","last_active":"2024-09-18T11:54:03.313823+0000","last_peered":"2024-09-18T11:54:03.313823+0000","last_clean":"2024-09-18T11:54:03.313823+0000","last_became_active":"2024-09-18T11:54:03.313417+0000","last_became_peered":"2024-09-18T11:54:03.313417+0000","last_unstale":"2024-09-18T11:54:03.313823+0000","last_undegraded":"2024-09-18T11:54:03.313823+0000","last_fullsized":"2024-09-18T11:54:03.313823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:33:53.902951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'4","reported_seq":27,"reported_epoch":24,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:03.313651+0000","last_change":"2024-09-18T11:54:03.313849+0000","last_active":"2024-09-18T11:54:03.313651+0000","last_peered":"2024-09-18T11:54:03.313651+0000","last_clean":"2024-09-18T11:53:53.108530+0000","last_became_active":"2024-09-18T11:54:03.313364+0000","last_became_peered":"2024-09-18T11:54:03.313364+0000","last_unstale":"2024-09-18T11:54:03.313651+0000","last_undegraded":"2024-09-18T11:54:03.300136+0000","last_fullsized":"2024-09-18T11:54:03.313651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:19:10.721166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":45,"reported_epoch":24,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:03.312546+0000","last_change":"2024-09-18T11:54:03.312725+0000","last_active":"2024-09-18T11:54:03.312546+0000","last_peered":"2024-09-18T11:54:03.312546+0000","last_clean":"2024-09-18T11:53:53.677796+0000","last_became_active":"2024-09-18T11:54:03.312454+0000","last_became_peered":"2024-09-18T11:54:03.312454+0000","last_unstale":"2024-09-18T11:54:03.312546+0000","last_undegraded":"2024-09-18T11:54:03.304337+0000","last_fullsized":"2024-09-18T11:54:03.312546+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:00:42.890017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.314267+0000","last_change":"2024-09-18T11:54:03.314267+0000","last_active":"2024-09-18T11:54:03.314267+0000","last_peered":"2024-09-18T11:54:03.314267+0000","last_clean":"2024-09-18T11:54:03.314267+0000","last_became_active":"2024-09-18T11:54:03.313170+0000","last_became_peered":"2024-09-18T11:54:03.313170+0000","last_unstale":"2024-09-18T11:54:03.314267+0000","last_undegraded":"2024-09-18T11:54:03.314267+0000","last_fullsized":"2024-09-18T11:54:03.314267+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:02:48.756542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"21'5","reported_seq":36,"reported_epoch":24,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:03.313472+0000","last_change":"2024-09-18T11:54:03.313718+0000","last_active":"2024-09-18T11:54:03.313472+0000","last_peered":"2024-09-18T11:54:03.313472+0000","last_clean":"2024-09-18T11:53:53.013443+0000","last_became_active":"2024-09-18T11:54:03.313342+0000","last_became_peered":"2024-09-18T11:54:03.313342+0000","last_unstale":"2024-09-18T11:54:03.313472+0000","last_undegraded":"2024-09-18T11:54:03.299898+0000","last_fullsized":"2024-09-18T11:54:03.313472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:10:26.570857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":34,"reported_epoch":24,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:03.312059+0000","last_change":"2024-09-18T11:54:03.312282+0000","last_active":"2024-09-18T11:54:03.312059+0000","last_peered":"2024-09-18T11:54:03.312059+0000","last_clean":"2024-09-18T11:53:53.869066+0000","last_became_active":"2024-09-18T11:54:03.311974+0000","last_became_peered":"2024-09-18T11:54:03.311974+0000","last_unstale":"2024-09-18T11:54:03.312059+0000","last_undegraded":"2024-09-18T11:54:03.303433+0000","last_fullsized":"2024-09-18T11:54:03.312059+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:10:29.718476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"21'4","reported_seq":35,"reported_epoch":24,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:03.311770+0000","last_change":"2024-09-18T11:54:03.313046+0000","last_active":"2024-09-18T11:54:03.311770+0000","last_peered":"2024-09-18T11:54:03.311770+0000","last_clean":"2024-09-18T11:53:53.251624+0000","last_became_active":"2024-09-18T11:54:03.311556+0000","last_became_peered":"2024-09-18T11:54:03.311556+0000","last_unstale":"2024-09-18T11:54:03.311770+0000","last_undegraded":"2024-09-18T11:54:03.302763+0000","last_fullsized":"2024-09-18T11:54:03.311770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:10:50.142128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"21'1","reported_seq":34,"reported_epoch":24,"state":"active+recovering+degraded","last_fresh":"2024-09-18T11:54:03.312462+0000","last_change":"2024-09-18T11:54:03.315441+0000","last_active":"2024-09-18T11:54:03.312462+0000","last_peered":"2024-09-18T11:54:03.312462+0000","last_clean":"2024-09-18T11:53:43.208142+0000","last_became_active":"2024-09-18T11:54:03.312204+0000","last_became_peered":"2024-09-18T11:54:03.312204+0000","last_unstale":"2024-09-18T11:54:03.312462+0000","last_undegraded":"2024-09-18T11:54:03.298908+0000","last_fullsized":"2024-09-18T11:54:03.312462+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:33:27.570280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.309168+0000","last_change":"2024-09-18T11:54:03.309168+0000","last_active":"2024-09-18T11:54:03.309168+0000","last_peered":"2024-09-18T11:54:03.309168+0000","last_clean":"2024-09-18T11:54:03.309168+0000","last_became_active":"2024-09-18T11:54:03.308772+0000","last_became_peered":"2024-09-18T11:54:03.308772+0000","last_unstale":"2024-09-18T11:54:03.309168+0000","last_undegraded":"2024-09-18T11:54:03.309168+0000","last_fullsized":"2024-09-18T11:54:03.309168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:47:51.845259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"21'5","reported_seq":48,"reported_epoch":24,"state":"active+recovering+degraded","last_fresh":"2024-09-18T11:54:03.311924+0000","last_change":"2024-09-18T11:54:03.315108+0000","last_active":"2024-09-18T11:54:03.311924+0000","last_peered":"2024-09-18T11:54:03.311924+0000","last_clean":"2024-09-18T11:53:53.726368+0000","last_became_active":"2024-09-18T11:54:03.311543+0000","last_became_peered":"2024-09-18T11:54:03.311543+0000","last_unstale":"2024-09-18T11:54:03.311924+0000","last_undegraded":"2024-09-18T11:54:03.303134+0000","last_fullsized":"2024-09-18T11:54:03.311924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:26:22.553547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.314466+0000","last_change":"2024-09-18T11:54:03.314466+0000","last_active":"2024-09-18T11:54:03.314466+0000","last_peered":"2024-09-18T11:54:03.314466+0000","last_clean":"2024-09-18T11:54:03.314466+0000","last_became_active":"2024-09-18T11:54:03.314137+0000","last_became_peered":"2024-09-18T11:54:03.314137+0000","last_unstale":"2024-09-18T11:54:03.314466+0000","last_undegraded":"2024-09-18T11:54:03.314466+0000","last_fullsized":"2024-09-18T11:54:03.314466+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:21:52.124644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"21'6","reported_seq":26,"reported_epoch":24,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:03.313548+0000","last_change":"2024-09-18T11:54:03.314456+0000","last_active":"2024-09-18T11:54:03.313548+0000","last_peered":"2024-09-18T11:54:03.313548+0000","last_clean":"2024-09-18T11:53:53.913371+0000","last_became_active":"2024-09-18T11:54:03.313449+0000","last_became_peered":"2024-09-18T11:54:03.313449+0000","last_unstale":"2024-09-18T11:54:03.313548+0000","last_undegraded":"2024-09-18T11:54:03.299824+0000","last_fullsized":"2024-09-18T11:54:03.313548+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:44:41.670535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.309583+0000","last_change":"2024-09-18T11:54:03.309583+0000","last_active":"2024-09-18T11:54:03.309583+0000","last_peered":"2024-09-18T11:54:03.309583+0000","last_clean":"2024-09-18T11:54:03.309583+0000","last_became_active":"2024-09-18T11:54:03.309100+0000","last_became_peered":"2024-09-18T11:54:03.309100+0000","last_unstale":"2024-09-18T11:54:03.309583+0000","last_undegraded":"2024-09-18T11:54:03.309583+0000","last_fullsized":"2024-09-18T11:54:03.309583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:44:42.857775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'3","reported_seq":37,"reported_epoch":24,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:03.313388+0000","last_change":"2024-09-18T11:54:03.313548+0000","last_active":"2024-09-18T11:54:03.313388+0000","last_peered":"2024-09-18T11:54:03.313388+0000","last_clean":"2024-09-18T11:53:52.962496+0000","last_became_active":"2024-09-18T11:54:03.313304+0000","last_became_peered":"2024-09-18T11:54:03.313304+0000","last_unstale":"2024-09-18T11:54:03.313388+0000","last_undegraded":"2024-09-18T11:54:03.303507+0000","last_fullsized":"2024-09-18T11:54:03.313388+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:33:56.738375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":58,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.317445+0000","last_change":"2024-09-18T11:54:03.317445+0000","last_active":"2024-09-18T11:54:03.317445+0000","last_peered":"2024-09-18T11:54:03.317445+0000","last_clean":"2024-09-18T11:54:03.317445+0000","last_became_active":"2024-09-18T11:54:03.317059+0000","last_became_peered":"2024-09-18T11:54:03.317059+0000","last_unstale":"2024-09-18T11:54:03.317445+0000","last_undegraded":"2024-09-18T11:54:03.317445+0000","last_fullsized":"2024-09-18T11:54:03.317445+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:02:06.392240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":38,"reported_epoch":24,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:03.312632+0000","last_change":"2024-09-18T11:54:03.312792+0000","last_active":"2024-09-18T11:54:03.312632+0000","last_peered":"2024-09-18T11:54:03.312632+0000","last_clean":"2024-09-18T11:53:53.346427+0000","last_became_active":"2024-09-18T11:54:03.312543+0000","last_became_peered":"2024-09-18T11:54:03.312543+0000","last_unstale":"2024-09-18T11:54:03.312632+0000","last_undegraded":"2024-09-18T11:54:03.303024+0000","last_fullsized":"2024-09-18T11:54:03.312632+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:58:49.486547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"21'1","reported_seq":31,"reported_epoch":24,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:03.316906+0000","last_change":"2024-09-18T11:54:03.317087+0000","last_active":"2024-09-18T11:54:03.316906+0000","last_peered":"2024-09-18T11:54:03.316906+0000","last_clean":"2024-09-18T11:53:46.243258+0000","last_became_active":"2024-09-18T11:54:03.316816+0000","last_became_peered":"2024-09-18T11:54:03.316816+0000","last_unstale":"2024-09-18T11:54:03.316906+0000","last_undegraded":"2024-09-18T11:54:03.304040+0000","last_fullsized":"2024-09-18T11:54:03.316906+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:54:44.458703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.317401+0000","last_change":"2024-09-18T11:54:03.317401+0000","last_active":"2024-09-18T11:54:03.317401+0000","last_peered":"2024-09-18T11:54:03.317401+0000","last_clean":"2024-09-18T11:54:03.317401+0000","last_became_active":"2024-09-18T11:54:03.316727+0000","last_became_peered":"2024-09-18T11:54:03.316727+0000","last_unstale":"2024-09-18T11:54:03.317401+0000","last_undegraded":"2024-09-18T11:54:03.317401+0000","last_fullsized":"2024-09-18T11:54:03.317401+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:21:21.336491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'5","reported_seq":36,"reported_epoch":24,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:03.317182+0000","last_change":"2024-09-18T11:54:03.317371+0000","last_active":"2024-09-18T11:54:03.317182+0000","last_peered":"2024-09-18T11:54:03.317182+0000","last_clean":"2024-09-18T11:53:53.823053+0000","last_became_active":"2024-09-18T11:54:03.317087+0000","last_became_peered":"2024-09-18T11:54:03.317087+0000","last_unstale":"2024-09-18T11:54:03.317182+0000","last_undegraded":"2024-09-18T11:54:03.303866+0000","last_fullsized":"2024-09-18T11:54:03.317182+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:05:48.525151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.313099+0000","last_change":"2024-09-18T11:54:03.313099+0000","last_active":"2024-09-18T11:54:03.313099+0000","last_peered":"2024-09-18T11:54:03.313099+0000","last_clean":"2024-09-18T11:54:03.313099+0000","last_became_active":"2024-09-18T11:54:03.312772+0000","last_became_peered":"2024-09-18T11:54:03.312772+0000","last_unstale":"2024-09-18T11:54:03.313099+0000","last_undegraded":"2024-09-18T11:54:03.313099+0000","last_fullsized":"2024-09-18T11:54:03.313099+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:48:40.437877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"21'2","reported_seq":32,"reported_epoch":24,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:03.317126+0000","last_change":"2024-09-18T11:54:03.317280+0000","last_active":"2024-09-18T11:54:03.317126+0000","last_peered":"2024-09-18T11:54:03.317126+0000","last_clean":"2024-09-18T11:53:46.242858+0000","last_became_active":"2024-09-18T11:54:03.317019+0000","last_became_peered":"2024-09-18T11:54:03.317019+0000","last_unstale":"2024-09-18T11:54:03.317126+0000","last_undegraded":"2024-09-18T11:54:03.304591+0000","last_fullsized":"2024-09-18T11:54:03.317126+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:07:21.838948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.311726+0000","last_change":"2024-09-18T11:54:03.311726+0000","last_active":"2024-09-18T11:54:03.311726+0000","last_peered":"2024-09-18T11:54:03.311726+0000","last_clean":"2024-09-18T11:54:03.311726+0000","last_became_active":"2024-09-18T11:54:03.311241+0000","last_became_peered":"2024-09-18T11:54:03.311241+0000","last_unstale":"2024-09-18T11:54:03.311726+0000","last_undegraded":"2024-09-18T11:54:03.311726+0000","last_fullsized":"2024-09-18T11:54:03.311726+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:04:29.336079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":53,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-18T11:54:03.312361+0000","last_change":"2024-09-18T11:54:03.312361+0000","last_active":"2024-09-18T11:54:03.312361+0000","last_peered":"2024-09-18T11:54:03.312361+0000","last_clean":"2024-09-18T11:54:03.312361+0000","last_became_active":"2024-09-18T11:54:03.311926+0000","last_became_peered":"2024-09-18T11:54:03.311926+0000","last_unstale":"2024-09-18T11:54:03.312361+0000","last_undegraded":"2024-09-18T11:54:03.312361+0000","last_fullsized":"2024-09-18T11:54:03.312361+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:32:06.515765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"21'9","reported_seq":40,"reported_epoch":24,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:03.312029+0000","last_change":"2024-09-18T11:54:03.312215+0000","last_active":"2024-09-18T11:54:03.312029+0000","last_peered":"2024-09-18T11:54:03.312029+0000","last_clean":"2024-09-18T11:53:53.582527+0000","last_became_active":"2024-09-18T11:54:03.311923+0000","last_became_peered":"2024-09-18T11:54:03.311923+0000","last_unstale":"2024-09-18T11:54:03.312029+0000","last_undegraded":"2024-09-18T11:54:03.307048+0000","last_fullsized":"2024-09-18T11:54:03.312029+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:28:28.605255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"1","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":17276,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":23446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738382,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297676,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074164,"statfs":{"total":96636764160,"available":96331943936,"internally_reserved":0,"allocated":323584,"data_stored":77342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247811,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298056,"kb_used_data":244,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94073784,"statfs":{"total":96636764160,"available":96331554816,"internally_reserved":0,"allocated":249856,"data_stored":66236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:08.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:08.540 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:08.540 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27/60 objects degraded (45.000%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":48,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":24,"num_osds":3,"num_up_osds":2,"osd_up_since":1726660442,"num_in_osds":2,"osd_in_since":1726660425,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+degraded","count":12},{"state_name":"active+clean","count":10},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":24,"num_pools":2,"num_objects":30,"data_bytes":17295,"bytes_used":610029568,"bytes_avail":192663498752,"bytes_total":193273528320,"degraded_objects":27,"degraded_total":60,"degraded_ratio":0.45000000000000001},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-18T11:53:47.015854+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:54:08.560 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': 'c6912087-b15b-4223-a3df-3a06756ee775', 'health': {'status': 'HEALTH_WARN', 'checks': {'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 27/60 objects degraded (45.000%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 48, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 24, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1726660442, 'num_in_osds': 2, 'osd_in_since': 1726660425, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_wait+degraded', 'count': 12}, {'state_name': 'active+clean', 'count': 10}, {'state_name': 'active+recovering+degraded', 'count': 2}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 30, 'data_bytes': 17295, 'bytes_used': 610029568, 'bytes_avail': 192663498752, 'bytes_total': 193273528320, 'degraded_objects': 27, 'degraded_total': 60, 'degraded_ratio': 0.45}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 2, 'modified': '2024-09-18T11:53:47.015854+0000', 'services': {'mgr': {'daemons': {'summary': '', 'x': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'mon': {'daemons': {'summary': '', 'a': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'b': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'c': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '1': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-18T11:54:08.560 INFO:tasks.rep_lost_unfound_delete.ceph_manager:active! 2024-09-18T11:54:08.560 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-18T11:54:08.560 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T11:54:08.624 INFO:tasks.ceph.osd.1:Stopped 2024-09-18T11:54:08.624 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-09-18T11:54:08.918 INFO:tasks.ceph.mon.a.smithi154.stderr:2024-09-18T11:54:08.913+0000 7f2c7dcba700 -1 mon.a@0(leader).osd e24 definitely_dead 0 2024-09-18T11:54:09.325 INFO:teuthology.orchestra.run.smithi154.stderr:marked down osd.1. 2024-09-18T11:54:09.336 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-09-18T11:54:09.631 INFO:tasks.ceph.mon.a.smithi154.stderr:2024-09-18T11:54:09.625+0000 7f2c7dcba700 -1 mon.a@0(leader).osd e25 definitely_dead 0 2024-09-18T11:54:10.331 INFO:teuthology.orchestra.run.smithi154.stderr:marked out osd.1. 2024-09-18T11:54:10.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd lost 1 --yes-i-really-mean-it 2024-09-18T11:54:11.335 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.329+0000 7f61e0092700 -1 log_channel(cluster) log [ERR] : 2.0 has 2 objects unfound and apparently lost 2024-09-18T11:54:11.337 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.329+0000 7f61df891700 -1 log_channel(cluster) log [ERR] : 2.1 has 2 objects unfound and apparently lost 2024-09-18T11:54:11.337 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.329+0000 7f61e0092700 -1 log_channel(cluster) log [ERR] : 2.8 has 1 objects unfound and apparently lost 2024-09-18T11:54:11.337 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.329+0000 7f61df891700 -1 log_channel(cluster) log [ERR] : 2.9 has 2 objects unfound and apparently lost 2024-09-18T11:54:11.337 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.329+0000 7f61df090700 -1 log_channel(cluster) log [ERR] : 2.a has 2 objects unfound and apparently lost 2024-09-18T11:54:11.337 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.329+0000 7f61e3098700 -1 log_channel(cluster) log [ERR] : 2.2 has 2 objects unfound and apparently lost 2024-09-18T11:54:11.337 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.329+0000 7f61e2897700 -1 log_channel(cluster) log [ERR] : 2.b has 2 objects unfound and apparently lost 2024-09-18T11:54:11.337 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.329+0000 7f61de08e700 -1 log_channel(cluster) log [ERR] : 2.4 has 3 objects unfound and apparently lost 2024-09-18T11:54:11.337 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.329+0000 7f61de88f700 -1 log_channel(cluster) log [ERR] : 2.3 has 2 objects unfound and apparently lost 2024-09-18T11:54:11.337 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.329+0000 7f61dd08c700 -1 log_channel(cluster) log [ERR] : 2.e has 2 objects unfound and apparently lost 2024-09-18T11:54:11.337 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.333+0000 7f61e1895700 -1 log_channel(cluster) log [ERR] : 2.d has 1 objects unfound and apparently lost 2024-09-18T11:54:11.337 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.333+0000 7f61dd88d700 -1 log_channel(cluster) log [ERR] : 2.5 has 2 objects unfound and apparently lost 2024-09-18T11:54:11.338 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.333+0000 7f61e1094700 -1 log_channel(cluster) log [ERR] : 2.6 has 3 objects unfound and apparently lost 2024-09-18T11:54:11.338 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:11.333+0000 7f61dc88b700 -1 log_channel(cluster) log [ERR] : 2.7 has 1 objects unfound and apparently lost 2024-09-18T11:54:11.340 INFO:teuthology.orchestra.run.smithi154.stderr:marked osd lost in epoch 25 2024-09-18T11:54:11.352 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-18T11:54:11.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-18T11:54:11.354 INFO:tasks.ceph.osd.2:Started 2024-09-18T11:54:11.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T11:54:11.449 INFO:teuthology.orchestra.run.smithi154.stderr:no valid command found; 10 closest matches: 2024-09-18T11:54:11.449 INFO:teuthology.orchestra.run.smithi154.stderr:0 2024-09-18T11:54:11.449 INFO:teuthology.orchestra.run.smithi154.stderr:1 2024-09-18T11:54:11.449 INFO:teuthology.orchestra.run.smithi154.stderr:2 2024-09-18T11:54:11.449 INFO:teuthology.orchestra.run.smithi154.stderr:abort 2024-09-18T11:54:11.449 INFO:teuthology.orchestra.run.smithi154.stderr:assert 2024-09-18T11:54:11.449 INFO:teuthology.orchestra.run.smithi154.stderr:config diff 2024-09-18T11:54:11.449 INFO:teuthology.orchestra.run.smithi154.stderr:config diff get 2024-09-18T11:54:11.449 INFO:teuthology.orchestra.run.smithi154.stderr:config get 2024-09-18T11:54:11.449 INFO:teuthology.orchestra.run.smithi154.stderr:config help [] 2024-09-18T11:54:11.449 INFO:teuthology.orchestra.run.smithi154.stderr:config set ... 2024-09-18T11:54:11.450 INFO:teuthology.orchestra.run.smithi154.stderr:admin_socket: invalid command 2024-09-18T11:54:11.450 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T11:54:11.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-18T11:54:11.958 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-18T11:54:11.953+0000 7f329fb02540 -1 Falling back to public interface 2024-09-18T11:54:14.570 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-18T11:54:14.564+0000 7f329fb02540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T11:54:15.361 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-18T11:54:15.356+0000 7f329fb02540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T11:54:15.699 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-18T11:54:15.692+0000 7f329fb02540 -1 osd.2 15 log_to_monitors true 2024-09-18T11:54:16.403 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.396+0000 7f61e0092700 -1 log_channel(cluster) log [ERR] : 2.0 has 2 objects unfound and apparently lost 2024-09-18T11:54:16.403 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.396+0000 7f61df891700 -1 log_channel(cluster) log [ERR] : 2.1 has 2 objects unfound and apparently lost 2024-09-18T11:54:16.404 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.396+0000 7f61e409a700 -1 log_channel(cluster) log [ERR] : 2.8 has 1 objects unfound and apparently lost 2024-09-18T11:54:16.404 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.396+0000 7f61e3098700 -1 log_channel(cluster) log [ERR] : 2.a has 2 objects unfound and apparently lost 2024-09-18T11:54:16.404 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.396+0000 7f61df891700 -1 log_channel(cluster) log [ERR] : 2.9 has 2 objects unfound and apparently lost 2024-09-18T11:54:16.404 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.396+0000 7f61e3098700 -1 log_channel(cluster) log [ERR] : 2.2 has 2 objects unfound and apparently lost 2024-09-18T11:54:16.404 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.396+0000 7f61e2897700 -1 log_channel(cluster) log [ERR] : 2.3 has 2 objects unfound and apparently lost 2024-09-18T11:54:16.404 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.396+0000 7f61de88f700 -1 log_channel(cluster) log [ERR] : 2.b has 2 objects unfound and apparently lost 2024-09-18T11:54:16.404 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.396+0000 7f61e2096700 -1 log_channel(cluster) log [ERR] : 2.4 has 3 objects unfound and apparently lost 2024-09-18T11:54:16.405 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.396+0000 7f61dd08c700 -1 log_channel(cluster) log [ERR] : 2.e has 2 objects unfound and apparently lost 2024-09-18T11:54:16.405 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.400+0000 7f61e1094700 -1 log_channel(cluster) log [ERR] : 2.6 has 3 objects unfound and apparently lost 2024-09-18T11:54:16.405 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.400+0000 7f61e1895700 -1 log_channel(cluster) log [ERR] : 2.d has 1 objects unfound and apparently lost 2024-09-18T11:54:16.405 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.400+0000 7f61dd88d700 -1 log_channel(cluster) log [ERR] : 2.5 has 2 objects unfound and apparently lost 2024-09-18T11:54:16.405 INFO:tasks.ceph.osd.0.smithi154.stderr:2024-09-18T11:54:16.400+0000 7f61dc88b700 -1 log_channel(cluster) log [ERR] : 2.7 has 1 objects unfound and apparently lost 2024-09-18T11:54:16.413 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-18T11:54:16.408+0000 7f3296049700 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:54:16.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T11:54:16.562 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:16.562 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-18T11:54:16.562 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-18T11:54:16.562 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:16.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2024-09-18T11:54:16.866 INFO:tasks.ceph.mon.a.smithi154.stderr:2024-09-18T11:54:16.860+0000 7f2c7dcba700 -1 mon.a@0(leader).osd e28 definitely_dead 0 2024-09-18T11:54:17.432 INFO:teuthology.orchestra.run.smithi154.stderr:marked in osd.2. 2024-09-18T11:54:17.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for osd.2 to be up 2024-09-18T11:54:17.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:54:17.724 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:17.724 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":29,"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","created":"2024-09-18T11:53:20.028520+0000","modified":"2024-09-18T11:54:17.405238+0000","last_up_change":"2024-09-18T11:54:17.405238+0000","last_in_change":"2024-09-18T11:54:17.405238+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:53:29.033564+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-09-18T11:53:40.807258+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":19,"up_from":23,"up_thru":25,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":26420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":26420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":26420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":26420}]},"public_addr":"172.21.15.154:6800/26420","cluster_addr":"172.21.15.154:6801/26420","heartbeat_back_addr":"172.21.15.154:6803/26420","heartbeat_front_addr":"172.21.15.154:6802/26420","state":["exists","up"]},{"osd":1,"uuid":"fd6ca4f9-93c5-4509-8fea-0121afa900f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":23,"down_at":25,"lost_at":25,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":21134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":21134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":21134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":21134}]},"public_addr":"172.21.15.154:6804/21134","cluster_addr":"172.21.15.154:6805/21134","heartbeat_back_addr":"172.21.15.154:6807/21134","heartbeat_front_addr":"172.21.15.154:6806/21134","state":["exists"]},{"osd":2,"uuid":"1554e072-019d-4d8e-b2ee-c0aac5a5e811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":15,"up_from":29,"up_thru":14,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":27666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":27666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":27666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":27666}]},"public_addr":"172.21.15.154:6804/27666","cluster_addr":"172.21.15.154:6805/27666","heartbeat_back_addr":"172.21.15.154:6807/27666","heartbeat_front_addr":"172.21.15.154:6806/27666","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.685684+0000","dead_epoch":20},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-18T11:53:26.545391+0000","dead_epoch":25},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.679601+0000","dead_epoch":17}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:54:17.737 INFO:tasks.rep_lost_unfound_delete.ceph_manager:osd.2 is up 2024-09-18T11:54:17.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:54:17.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T11:54:17.891 INFO:teuthology.orchestra.run.smithi154.stdout:124554051587 2024-09-18T11:54:17.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:54:17.895 INFO:teuthology.orchestra.run.smithi154.stdout:98784247814 2024-09-18T11:54:17.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:54:18.180 INFO:teuthology.orchestra.run.smithi154.stdout:0 2024-09-18T11:54:18.193 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 124554051587 got 0 for osd.2 2024-09-18T11:54:18.216 INFO:teuthology.orchestra.run.smithi154.stdout:98784247813 2024-09-18T11:54:18.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 98784247814 got 98784247813 for osd.0 2024-09-18T11:54:19.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:54:19.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:54:19.492 INFO:teuthology.orchestra.run.smithi154.stdout:124554051587 2024-09-18T11:54:19.506 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 124554051587 got 124554051587 for osd.2 2024-09-18T11:54:19.506 DEBUG:teuthology.parallel:result is None 2024-09-18T11:54:19.517 INFO:teuthology.orchestra.run.smithi154.stdout:98784247814 2024-09-18T11:54:19.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 98784247814 got 98784247814 for osd.0 2024-09-18T11:54:19.536 DEBUG:teuthology.parallel:result is None 2024-09-18T11:54:19.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting till active 2024-09-18T11:54:19.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:19.819 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:19.824 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:54:19.838 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-09-18T11:54:19.025863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17295,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596204,"kb_used_data":524,"kb_used_omap":8,"kb_used_meta":595639,"kb_avail":188147476,"statfs":{"total":193273528320,"available":192663015424,"internally_reserved":0,"allocated":536576,"data_stored":157231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9204,"internal_metadata":609934348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-57,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002316"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":34,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.421834+0000","last_change":"2024-09-18T11:54:17.424599+0000","last_active":"2024-09-18T11:54:17.421805+0000","last_peered":"2024-09-18T11:54:16.404148+0000","last_clean":"2024-09-18T11:54:03.317537+0000","last_became_active":"2024-09-18T11:54:10.325667+0000","last_became_peered":"2024-09-18T11:54:10.325667+0000","last_unstale":"2024-09-18T11:54:17.421834+0000","last_undegraded":"2024-09-18T11:54:17.421834+0000","last_fullsized":"2024-09-18T11:54:17.421834+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:10:14.458618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'4","reported_seq":13,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:16.403301+0000","last_change":"2024-09-18T11:54:17.429543+0000","last_active":"2024-09-18T11:54:17.429543+0000","last_peered":"2024-09-18T11:54:16.403301+0000","last_clean":"2024-09-18T11:53:53.108530+0000","last_became_active":"2024-09-18T11:54:10.323218+0000","last_became_peered":"2024-09-18T11:54:10.323218+0000","last_unstale":"2024-09-18T11:54:16.403301+0000","last_undegraded":"2024-09-18T11:54:10.319427+0000","last_fullsized":"2024-09-18T11:54:10.319090+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:31:16.015386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":7,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:16.403683+0000","last_change":"2024-09-18T11:54:17.430173+0000","last_active":"2024-09-18T11:54:17.430173+0000","last_peered":"2024-09-18T11:54:16.403683+0000","last_clean":"2024-09-18T11:53:53.677796+0000","last_became_active":"2024-09-18T11:54:10.325297+0000","last_became_peered":"2024-09-18T11:54:10.325297+0000","last_unstale":"2024-09-18T11:54:16.403683+0000","last_undegraded":"2024-09-18T11:54:10.321642+0000","last_fullsized":"2024-09-18T11:54:10.321398+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:28:09.678683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.426358+0000","last_change":"2024-09-18T11:54:17.414741+0000","last_active":"2024-09-18T11:54:17.414195+0000","last_peered":"2024-09-18T11:53:42.205446+0000","last_clean":"2024-09-18T11:53:42.205446+0000","last_became_active":"2024-09-18T11:53:42.204788+0000","last_became_peered":"2024-09-18T11:53:42.204788+0000","last_unstale":"2024-09-18T11:54:17.426358+0000","last_undegraded":"2024-09-18T11:54:17.426358+0000","last_fullsized":"2024-09-18T11:54:17.426358+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:47:33.514639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"21'5","reported_seq":49,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.422017+0000","last_change":"2024-09-18T11:54:17.424582+0000","last_active":"2024-09-18T11:54:17.421984+0000","last_peered":"2024-09-18T11:54:16.402828+0000","last_clean":"2024-09-18T11:53:53.013443+0000","last_became_active":"2024-09-18T11:54:10.323785+0000","last_became_peered":"2024-09-18T11:54:10.323785+0000","last_unstale":"2024-09-18T11:54:17.422017+0000","last_undegraded":"2024-09-18T11:54:17.422017+0000","last_fullsized":"2024-09-18T11:54:17.422017+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:00:36.147746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":42,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.416170+0000","last_change":"2024-09-18T11:54:17.416313+0000","last_active":"2024-09-18T11:54:17.416133+0000","last_peered":"2024-09-18T11:54:16.401685+0000","last_clean":"2024-09-18T11:53:53.869066+0000","last_became_active":"2024-09-18T11:54:10.323003+0000","last_became_peered":"2024-09-18T11:54:10.323003+0000","last_unstale":"2024-09-18T11:54:17.416170+0000","last_undegraded":"2024-09-18T11:54:17.416170+0000","last_fullsized":"2024-09-18T11:54:17.416170+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:58:17.395800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"21'4","reported_seq":43,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.421345+0000","last_change":"2024-09-18T11:54:17.421588+0000","last_active":"2024-09-18T11:54:17.421291+0000","last_peered":"2024-09-18T11:54:16.401705+0000","last_clean":"2024-09-18T11:53:53.251624+0000","last_became_active":"2024-09-18T11:54:10.323146+0000","last_became_peered":"2024-09-18T11:54:10.323146+0000","last_unstale":"2024-09-18T11:54:17.421345+0000","last_undegraded":"2024-09-18T11:54:17.421345+0000","last_fullsized":"2024-09-18T11:54:17.421345+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:31:10.882855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'1","reported_seq":47,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.416052+0000","last_change":"2024-09-18T11:54:17.416312+0000","last_active":"2024-09-18T11:54:17.416015+0000","last_peered":"2024-09-18T11:54:16.401279+0000","last_clean":"2024-09-18T11:53:43.208142+0000","last_became_active":"2024-09-18T11:54:10.320792+0000","last_became_peered":"2024-09-18T11:54:10.320792+0000","last_unstale":"2024-09-18T11:54:17.416052+0000","last_undegraded":"2024-09-18T11:54:17.416052+0000","last_fullsized":"2024-09-18T11:54:17.416052+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:09:33.185238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.414073+0000","last_change":"2024-09-18T11:54:17.414635+0000","last_active":"2024-09-18T11:54:17.413965+0000","last_peered":"2024-09-18T11:54:16.400819+0000","last_clean":"2024-09-18T11:53:46.242383+0000","last_became_active":"2024-09-18T11:54:10.320174+0000","last_became_peered":"2024-09-18T11:54:10.320174+0000","last_unstale":"2024-09-18T11:54:17.414073+0000","last_undegraded":"2024-09-18T11:54:17.414073+0000","last_fullsized":"2024-09-18T11:54:17.414073+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:41:27.726077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"21'5","reported_seq":7,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:16.402811+0000","last_change":"2024-09-18T11:54:17.430217+0000","last_active":"2024-09-18T11:54:17.430217+0000","last_peered":"2024-09-18T11:54:16.402811+0000","last_clean":"2024-09-18T11:53:53.726368+0000","last_became_active":"2024-09-18T11:54:10.323491+0000","last_became_peered":"2024-09-18T11:54:10.323491+0000","last_unstale":"2024-09-18T11:54:16.402811+0000","last_undegraded":"2024-09-18T11:54:10.319504+0000","last_fullsized":"2024-09-18T11:54:10.319193+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:11:21.352233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.422037+0000","last_change":"2024-09-18T11:54:17.417244+0000","last_active":"2024-09-18T11:54:16.410702+0000","last_peered":"2024-09-18T11:53:40.769374+0000","last_clean":"2024-09-18T11:53:40.769374+0000","last_became_active":"2024-09-18T11:53:30.112377+0000","last_became_peered":"2024-09-18T11:53:30.112377+0000","last_unstale":"2024-09-18T11:54:17.422037+0000","last_undegraded":"2024-09-18T11:54:17.422037+0000","last_fullsized":"2024-09-18T11:54:17.422037+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:11:14.773795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000428744,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"21'6","reported_seq":39,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.417268+0000","last_change":"2024-09-18T11:54:17.417512+0000","last_active":"2024-09-18T11:54:17.417234+0000","last_peered":"2024-09-18T11:54:16.402186+0000","last_clean":"2024-09-18T11:53:53.913371+0000","last_became_active":"2024-09-18T11:54:10.323425+0000","last_became_peered":"2024-09-18T11:54:10.323425+0000","last_unstale":"2024-09-18T11:54:17.417268+0000","last_undegraded":"2024-09-18T11:54:17.417268+0000","last_fullsized":"2024-09-18T11:54:17.417268+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:29.643522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":45,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.421829+0000","last_change":"2024-09-18T11:54:17.422099+0000","last_active":"2024-09-18T11:54:17.421802+0000","last_peered":"2024-09-18T11:54:16.401062+0000","last_clean":"2024-09-18T11:54:03.316651+0000","last_became_active":"2024-09-18T11:54:10.322927+0000","last_became_peered":"2024-09-18T11:54:10.322927+0000","last_unstale":"2024-09-18T11:54:17.421829+0000","last_undegraded":"2024-09-18T11:54:17.421829+0000","last_fullsized":"2024-09-18T11:54:17.421829+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:59:54.412456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'3","reported_seq":13,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:16.401182+0000","last_change":"2024-09-18T11:54:17.429644+0000","last_active":"2024-09-18T11:54:17.429644+0000","last_peered":"2024-09-18T11:54:16.401182+0000","last_clean":"2024-09-18T11:53:52.962496+0000","last_became_active":"2024-09-18T11:54:10.321981+0000","last_became_peered":"2024-09-18T11:54:10.321981+0000","last_unstale":"2024-09-18T11:54:16.401182+0000","last_undegraded":"2024-09-18T11:54:10.318368+0000","last_fullsized":"2024-09-18T11:54:10.318036+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:57:40.116608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":13,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.423927+0000","last_change":"2024-09-18T11:54:17.417473+0000","last_active":"2024-09-18T11:54:17.417244+0000","last_peered":"2024-09-18T11:53:30.115248+0000","last_clean":"2024-09-18T11:53:30.115248+0000","last_became_active":"2024-09-18T11:53:30.112162+0000","last_became_peered":"2024-09-18T11:53:30.112162+0000","last_unstale":"2024-09-18T11:54:17.423927+0000","last_undegraded":"2024-09-18T11:54:17.423927+0000","last_fullsized":"2024-09-18T11:54:17.423927+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:09.616062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":13,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:16.400947+0000","last_change":"2024-09-18T11:54:17.430057+0000","last_active":"2024-09-18T11:54:17.430057+0000","last_peered":"2024-09-18T11:54:16.400947+0000","last_clean":"2024-09-18T11:53:53.346427+0000","last_became_active":"2024-09-18T11:54:10.320064+0000","last_became_peered":"2024-09-18T11:54:10.320064+0000","last_unstale":"2024-09-18T11:54:16.400947+0000","last_undegraded":"2024-09-18T11:54:10.316959+0000","last_fullsized":"2024-09-18T11:54:10.316620+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:56:18.400692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"21'1","reported_seq":39,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.419247+0000","last_change":"2024-09-18T11:54:17.421682+0000","last_active":"2024-09-18T11:54:17.419080+0000","last_peered":"2024-09-18T11:54:16.404137+0000","last_clean":"2024-09-18T11:53:46.243258+0000","last_became_active":"2024-09-18T11:54:10.325116+0000","last_became_peered":"2024-09-18T11:54:10.325116+0000","last_unstale":"2024-09-18T11:54:17.419247+0000","last_undegraded":"2024-09-18T11:54:17.419247+0000","last_fullsized":"2024-09-18T11:54:17.419247+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:47:50.891364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":48,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.421207+0000","last_change":"2024-09-18T11:54:17.421502+0000","last_active":"2024-09-18T11:54:17.421164+0000","last_peered":"2024-09-18T11:54:16.403376+0000","last_clean":"2024-09-18T11:53:46.242675+0000","last_became_active":"2024-09-18T11:54:10.324761+0000","last_became_peered":"2024-09-18T11:54:10.324761+0000","last_unstale":"2024-09-18T11:54:17.421207+0000","last_undegraded":"2024-09-18T11:54:17.421207+0000","last_fullsized":"2024-09-18T11:54:17.421207+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:13:41.344635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"21'5","reported_seq":44,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.421790+0000","last_change":"2024-09-18T11:54:17.424643+0000","last_active":"2024-09-18T11:54:17.421756+0000","last_peered":"2024-09-18T11:54:16.403525+0000","last_clean":"2024-09-18T11:53:53.823053+0000","last_became_active":"2024-09-18T11:54:10.324934+0000","last_became_peered":"2024-09-18T11:54:10.324934+0000","last_unstale":"2024-09-18T11:54:17.421790+0000","last_undegraded":"2024-09-18T11:54:17.421790+0000","last_fullsized":"2024-09-18T11:54:17.421790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:02:57.682768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.423054+0000","last_change":"2024-09-18T11:54:17.418838+0000","last_active":"2024-09-18T11:54:16.410717+0000","last_peered":"2024-09-18T11:53:40.769540+0000","last_clean":"2024-09-18T11:53:40.769540+0000","last_became_active":"2024-09-18T11:53:30.112318+0000","last_became_peered":"2024-09-18T11:53:30.112318+0000","last_unstale":"2024-09-18T11:54:17.423054+0000","last_undegraded":"2024-09-18T11:54:17.423054+0000","last_fullsized":"2024-09-18T11:54:17.423054+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:38:17.631603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047085699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'2","reported_seq":40,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.421956+0000","last_change":"2024-09-18T11:54:17.422188+0000","last_active":"2024-09-18T11:54:17.421920+0000","last_peered":"2024-09-18T11:54:16.403803+0000","last_clean":"2024-09-18T11:53:46.242858+0000","last_became_active":"2024-09-18T11:54:10.325085+0000","last_became_peered":"2024-09-18T11:54:10.325085+0000","last_unstale":"2024-09-18T11:54:17.421956+0000","last_undegraded":"2024-09-18T11:54:17.421956+0000","last_fullsized":"2024-09-18T11:54:17.421956+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:07:33.911727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":48,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.422166+0000","last_change":"2024-09-18T11:54:17.422380+0000","last_active":"2024-09-18T11:54:17.422140+0000","last_peered":"2024-09-18T11:54:16.403021+0000","last_clean":"2024-09-18T11:53:46.243252+0000","last_became_active":"2024-09-18T11:54:10.322863+0000","last_became_peered":"2024-09-18T11:54:10.322863+0000","last_unstale":"2024-09-18T11:54:17.422166+0000","last_undegraded":"2024-09-18T11:54:17.422166+0000","last_fullsized":"2024-09-18T11:54:17.422166+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:11:33.327030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":6,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.423033+0000","last_change":"2024-09-18T11:54:17.417172+0000","last_active":"2024-09-18T11:53:29.091408+0000","last_peered":"2024-09-18T11:53:29.091408+0000","last_clean":"2024-09-18T11:53:29.091408+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T11:54:17.423033+0000","last_undegraded":"2024-09-18T11:54:17.423033+0000","last_fullsized":"2024-09-18T11:54:17.423033+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:50:40.027145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"21'9","reported_seq":48,"reported_epoch":29,"state":"peering","last_fresh":"2024-09-18T11:54:17.418708+0000","last_change":"2024-09-18T11:54:17.422236+0000","last_active":"2024-09-18T11:54:17.418668+0000","last_peered":"2024-09-18T11:54:16.402950+0000","last_clean":"2024-09-18T11:53:53.582527+0000","last_became_active":"2024-09-18T11:54:10.324008+0000","last_became_peered":"2024-09-18T11:54:10.324008+0000","last_unstale":"2024-09-18T11:54:17.418708+0000","last_undegraded":"2024-09-18T11:54:17.418708+0000","last_fullsized":"2024-09-18T11:54:17.418708+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:30:04.282662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":17276,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":23446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051587,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298112,"kb_used_data":240,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073728,"statfs":{"total":96636764160,"available":96331497472,"internally_reserved":0,"allocated":245760,"data_stored":75521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247814,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298092,"kb_used_data":284,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94073748,"statfs":{"total":96636764160,"available":96331517952,"internally_reserved":0,"allocated":290816,"data_stored":81710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:19.839 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:20.170 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:20.170 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":60,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":30,"num_osds":3,"num_up_osds":2,"osd_up_since":1726660457,"num_in_osds":2,"osd_in_since":1726660457,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":24}],"num_pgs":24,"num_pools":2,"num_objects":30,"data_bytes":17295,"bytes_used":610512896,"bytes_avail":192663015424,"bytes_total":193273528320,"inactive_pgs_ratio":1},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-09-18T11:54:11.022915+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:54:20.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': 'c6912087-b15b-4223-a3df-3a06756ee775', 'health': {'status': 'HEALTH_OK', 'checks': {}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 60, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 30, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1726660457, 'num_in_osds': 2, 'osd_in_since': 1726660457, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'peering', 'count': 24}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 30, 'data_bytes': 17295, 'bytes_used': 610512896, 'bytes_avail': 192663015424, 'bytes_total': 193273528320, 'inactive_pgs_ratio': 1}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 3, 'modified': '2024-09-18T11:54:11.022915+0000', 'services': {'mgr': {'daemons': {'summary': '', 'x': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'mon': {'daemons': {'summary': '', 'a': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'b': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'c': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-18T11:54:23.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:23.452 INFO:tasks.ceph.mon.a.smithi154.stderr:2024-09-18T11:54:23.448+0000 7f2c804bf700 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 5 pgs recovery_unfound (PG_DAMAGED) 2024-09-18T11:54:23.481 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:23.484 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:54:23.494 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2024-09-18T11:54:23.026991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":17295,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596228,"kb_used_data":540,"kb_used_omap":8,"kb_used_meta":595639,"kb_avail":188147452,"statfs":{"total":193273528320,"available":192662990848,"internally_reserved":0,"allocated":552960,"data_stored":162641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9204,"internal_metadata":609934348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.627475"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.444208+0000","last_change":"2024-09-18T11:54:18.444208+0000","last_active":"2024-09-18T11:54:18.444208+0000","last_peered":"2024-09-18T11:54:18.444208+0000","last_clean":"2024-09-18T11:54:18.444208+0000","last_became_active":"2024-09-18T11:54:18.443917+0000","last_became_peered":"2024-09-18T11:54:18.443917+0000","last_unstale":"2024-09-18T11:54:18.444208+0000","last_undegraded":"2024-09-18T11:54:18.444208+0000","last_fullsized":"2024-09-18T11:54:18.444208+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:52:17.496947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'4","reported_seq":21,"reported_epoch":30,"state":"active+recovery_unfound","last_fresh":"2024-09-18T11:54:18.446476+0000","last_change":"2024-09-18T11:54:18.446754+0000","last_active":"2024-09-18T11:54:18.446476+0000","last_peered":"2024-09-18T11:54:18.446476+0000","last_clean":"2024-09-18T11:53:53.108530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:54:18.446476+0000","last_undegraded":"2024-09-18T11:54:18.430669+0000","last_fullsized":"2024-09-18T11:54:18.446476+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:31:16.015386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":15,"reported_epoch":30,"state":"active+recovery_unfound","last_fresh":"2024-09-18T11:54:18.445197+0000","last_change":"2024-09-18T11:54:18.445446+0000","last_active":"2024-09-18T11:54:18.445197+0000","last_peered":"2024-09-18T11:54:18.445197+0000","last_clean":"2024-09-18T11:53:53.677796+0000","last_became_active":"2024-09-18T11:54:18.439908+0000","last_became_peered":"2024-09-18T11:54:18.439908+0000","last_unstale":"2024-09-18T11:54:18.445197+0000","last_undegraded":"2024-09-18T11:54:18.431194+0000","last_fullsized":"2024-09-18T11:54:18.445197+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:28:09.678683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439837+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:54:18.439837+0000","last_peered":"2024-09-18T11:54:18.439837+0000","last_clean":"2024-09-18T11:54:18.439837+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:54:18.439837+0000","last_undegraded":"2024-09-18T11:54:18.439837+0000","last_fullsized":"2024-09-18T11:54:18.439837+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:35.240042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"21'5","reported_seq":55,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.442082+0000","last_change":"2024-09-18T11:54:18.443818+0000","last_active":"2024-09-18T11:54:18.442082+0000","last_peered":"2024-09-18T11:54:18.442082+0000","last_clean":"2024-09-18T11:53:53.013443+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:54:18.442082+0000","last_undegraded":"2024-09-18T11:54:18.431175+0000","last_fullsized":"2024-09-18T11:54:18.442082+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:47:11.063973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":48,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.441365+0000","last_change":"2024-09-18T11:54:18.442338+0000","last_active":"2024-09-18T11:54:18.441365+0000","last_peered":"2024-09-18T11:54:18.441365+0000","last_clean":"2024-09-18T11:53:53.869066+0000","last_became_active":"2024-09-18T11:54:18.441285+0000","last_became_peered":"2024-09-18T11:54:18.441285+0000","last_unstale":"2024-09-18T11:54:18.441365+0000","last_undegraded":"2024-09-18T11:54:18.433470+0000","last_fullsized":"2024-09-18T11:54:18.441365+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:50:27.827107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"21'4","reported_seq":49,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.445858+0000","last_change":"2024-09-18T11:54:18.446054+0000","last_active":"2024-09-18T11:54:18.445858+0000","last_peered":"2024-09-18T11:54:18.445858+0000","last_clean":"2024-09-18T11:53:53.251624+0000","last_became_active":"2024-09-18T11:54:18.445771+0000","last_became_peered":"2024-09-18T11:54:18.445771+0000","last_unstale":"2024-09-18T11:54:18.445858+0000","last_undegraded":"2024-09-18T11:54:18.434522+0000","last_fullsized":"2024-09-18T11:54:18.445858+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:54:48.333477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'1","reported_seq":53,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.445868+0000","last_change":"2024-09-18T11:54:18.446046+0000","last_active":"2024-09-18T11:54:18.445868+0000","last_peered":"2024-09-18T11:54:18.445868+0000","last_clean":"2024-09-18T11:53:43.208142+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:54:18.445868+0000","last_undegraded":"2024-09-18T11:54:18.434814+0000","last_fullsized":"2024-09-18T11:54:18.445868+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:22.388387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441722+0000","last_change":"2024-09-18T11:54:18.441722+0000","last_active":"2024-09-18T11:54:18.441722+0000","last_peered":"2024-09-18T11:54:18.441722+0000","last_clean":"2024-09-18T11:54:18.441722+0000","last_became_active":"2024-09-18T11:54:18.440263+0000","last_became_peered":"2024-09-18T11:54:18.440263+0000","last_unstale":"2024-09-18T11:54:18.441722+0000","last_undegraded":"2024-09-18T11:54:18.441722+0000","last_fullsized":"2024-09-18T11:54:18.441722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:27:26.196691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"21'5","reported_seq":15,"reported_epoch":30,"state":"active+recovery_unfound","last_fresh":"2024-09-18T11:54:18.447139+0000","last_change":"2024-09-18T11:54:18.447311+0000","last_active":"2024-09-18T11:54:18.447139+0000","last_peered":"2024-09-18T11:54:18.447139+0000","last_clean":"2024-09-18T11:53:53.726368+0000","last_became_active":"2024-09-18T11:54:18.440195+0000","last_became_peered":"2024-09-18T11:54:18.440195+0000","last_unstale":"2024-09-18T11:54:18.447139+0000","last_undegraded":"2024-09-18T11:54:18.433786+0000","last_fullsized":"2024-09-18T11:54:18.447139+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:11:21.352233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439949+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:54:18.439949+0000","last_peered":"2024-09-18T11:54:18.439949+0000","last_clean":"2024-09-18T11:54:18.439949+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:54:18.439949+0000","last_undegraded":"2024-09-18T11:54:18.439949+0000","last_fullsized":"2024-09-18T11:54:18.439949+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:14:25.540910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"21'6","reported_seq":45,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.445793+0000","last_change":"2024-09-18T11:54:18.445976+0000","last_active":"2024-09-18T11:54:18.445793+0000","last_peered":"2024-09-18T11:54:18.445793+0000","last_clean":"2024-09-18T11:53:53.913371+0000","last_became_active":"2024-09-18T11:54:18.445692+0000","last_became_peered":"2024-09-18T11:54:18.445692+0000","last_unstale":"2024-09-18T11:54:18.445793+0000","last_undegraded":"2024-09-18T11:54:18.433038+0000","last_fullsized":"2024-09-18T11:54:18.445793+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:45:22.401658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":51,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.441089+0000","last_change":"2024-09-18T11:54:18.441264+0000","last_active":"2024-09-18T11:54:18.441089+0000","last_peered":"2024-09-18T11:54:18.441089+0000","last_clean":"2024-09-18T11:54:03.316651+0000","last_became_active":"2024-09-18T11:54:18.440963+0000","last_became_peered":"2024-09-18T11:54:18.440963+0000","last_unstale":"2024-09-18T11:54:18.441089+0000","last_undegraded":"2024-09-18T11:54:18.432164+0000","last_fullsized":"2024-09-18T11:54:18.441089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:23:22.104360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'3","reported_seq":21,"reported_epoch":30,"state":"active+recovery_unfound","last_fresh":"2024-09-18T11:54:18.447684+0000","last_change":"2024-09-18T11:54:18.447897+0000","last_active":"2024-09-18T11:54:18.447684+0000","last_peered":"2024-09-18T11:54:18.447684+0000","last_clean":"2024-09-18T11:53:52.962496+0000","last_became_active":"2024-09-18T11:54:18.438876+0000","last_became_peered":"2024-09-18T11:54:18.438876+0000","last_unstale":"2024-09-18T11:54:18.447684+0000","last_undegraded":"2024-09-18T11:54:18.430632+0000","last_fullsized":"2024-09-18T11:54:18.447684+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:57:40.116608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439859+0000","last_change":"2024-09-18T11:54:18.439859+0000","last_active":"2024-09-18T11:54:18.439859+0000","last_peered":"2024-09-18T11:54:18.439859+0000","last_clean":"2024-09-18T11:54:18.439859+0000","last_became_active":"2024-09-18T11:54:18.438620+0000","last_became_peered":"2024-09-18T11:54:18.438620+0000","last_unstale":"2024-09-18T11:54:18.439859+0000","last_undegraded":"2024-09-18T11:54:18.439859+0000","last_fullsized":"2024-09-18T11:54:18.439859+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:09:25.244649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":21,"reported_epoch":30,"state":"active+recovery_unfound","last_fresh":"2024-09-18T11:54:18.448218+0000","last_change":"2024-09-18T11:54:18.448486+0000","last_active":"2024-09-18T11:54:18.448218+0000","last_peered":"2024-09-18T11:54:18.448218+0000","last_clean":"2024-09-18T11:53:53.346427+0000","last_became_active":"2024-09-18T11:54:18.441199+0000","last_became_peered":"2024-09-18T11:54:18.441199+0000","last_unstale":"2024-09-18T11:54:18.448218+0000","last_undegraded":"2024-09-18T11:54:18.430238+0000","last_fullsized":"2024-09-18T11:54:18.448218+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:56:18.400692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"21'1","reported_seq":45,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.445693+0000","last_change":"2024-09-18T11:54:18.445855+0000","last_active":"2024-09-18T11:54:18.445693+0000","last_peered":"2024-09-18T11:54:18.445693+0000","last_clean":"2024-09-18T11:53:46.243258+0000","last_became_active":"2024-09-18T11:54:18.445615+0000","last_became_peered":"2024-09-18T11:54:18.445615+0000","last_unstale":"2024-09-18T11:54:18.445693+0000","last_undegraded":"2024-09-18T11:54:18.433012+0000","last_fullsized":"2024-09-18T11:54:18.445693+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:33:38.875122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.442188+0000","last_change":"2024-09-18T11:54:18.442188+0000","last_active":"2024-09-18T11:54:18.442188+0000","last_peered":"2024-09-18T11:54:18.442188+0000","last_clean":"2024-09-18T11:54:18.442188+0000","last_became_active":"2024-09-18T11:54:18.441831+0000","last_became_peered":"2024-09-18T11:54:18.441831+0000","last_unstale":"2024-09-18T11:54:18.442188+0000","last_undegraded":"2024-09-18T11:54:18.442188+0000","last_fullsized":"2024-09-18T11:54:18.442188+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:20:27.929651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"21'5","reported_seq":50,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.441768+0000","last_change":"2024-09-18T11:54:18.441937+0000","last_active":"2024-09-18T11:54:18.441768+0000","last_peered":"2024-09-18T11:54:18.441768+0000","last_clean":"2024-09-18T11:53:53.823053+0000","last_became_active":"2024-09-18T11:54:18.441659+0000","last_became_peered":"2024-09-18T11:54:18.441659+0000","last_unstale":"2024-09-18T11:54:18.441768+0000","last_undegraded":"2024-09-18T11:54:18.432961+0000","last_fullsized":"2024-09-18T11:54:18.441768+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:12:36.428174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440580+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:54:18.440580+0000","last_peered":"2024-09-18T11:54:18.440580+0000","last_clean":"2024-09-18T11:54:18.440580+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:54:18.440580+0000","last_undegraded":"2024-09-18T11:54:18.440580+0000","last_fullsized":"2024-09-18T11:54:18.440580+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:10:36.497525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"21'2","reported_seq":47,"reported_epoch":30,"state":"active+recovering+degraded","last_fresh":"2024-09-18T11:54:18.441470+0000","last_change":"2024-09-18T11:54:18.446026+0000","last_active":"2024-09-18T11:54:18.441470+0000","last_peered":"2024-09-18T11:54:18.441470+0000","last_clean":"2024-09-18T11:53:46.242858+0000","last_became_active":"2024-09-18T11:54:18.440831+0000","last_became_peered":"2024-09-18T11:54:18.440831+0000","last_unstale":"2024-09-18T11:54:18.441470+0000","last_undegraded":"2024-09-18T11:54:18.431755+0000","last_fullsized":"2024-09-18T11:54:18.441470+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:18:26.703672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441579+0000","last_change":"2024-09-18T11:54:18.441579+0000","last_active":"2024-09-18T11:54:18.441579+0000","last_peered":"2024-09-18T11:54:18.441579+0000","last_clean":"2024-09-18T11:54:18.441579+0000","last_became_active":"2024-09-18T11:54:18.440490+0000","last_became_peered":"2024-09-18T11:54:18.440490+0000","last_unstale":"2024-09-18T11:54:18.441579+0000","last_undegraded":"2024-09-18T11:54:18.441579+0000","last_fullsized":"2024-09-18T11:54:18.441579+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:04.506685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440411+0000","last_change":"2024-09-18T11:54:18.440411+0000","last_active":"2024-09-18T11:54:18.440411+0000","last_peered":"2024-09-18T11:54:18.440411+0000","last_clean":"2024-09-18T11:54:18.440411+0000","last_became_active":"2024-09-18T11:54:18.439759+0000","last_became_peered":"2024-09-18T11:54:18.439759+0000","last_unstale":"2024-09-18T11:54:18.440411+0000","last_undegraded":"2024-09-18T11:54:18.440411+0000","last_fullsized":"2024-09-18T11:54:18.440411+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:02.939140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"21'9","reported_seq":54,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.440934+0000","last_change":"2024-09-18T11:54:18.441905+0000","last_active":"2024-09-18T11:54:18.440934+0000","last_peered":"2024-09-18T11:54:18.440934+0000","last_clean":"2024-09-18T11:53:53.582527+0000","last_became_active":"2024-09-18T11:54:18.440844+0000","last_became_peered":"2024-09-18T11:54:18.440844+0000","last_unstale":"2024-09-18T11:54:18.440934+0000","last_undegraded":"2024-09-18T11:54:18.431783+0000","last_fullsized":"2024-09-18T11:54:18.440934+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:38:24.322706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":17276,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":51,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":23446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051588,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298124,"kb_used_data":248,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073716,"statfs":{"total":96636764160,"available":96331485184,"internally_reserved":0,"allocated":253952,"data_stored":78226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247815,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298104,"kb_used_data":292,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94073736,"statfs":{"total":96636764160,"available":96331505664,"internally_reserved":0,"allocated":299008,"data_stored":84415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:23.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:23.844 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:23.844 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/30 objects unfound (90.000%)","count":27},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 5 pgs recovery_unfound","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 52/60 objects degraded (86.667%), 10 pgs degraded","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":63,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":30,"num_osds":3,"num_up_osds":2,"osd_up_since":1726660457,"num_in_osds":2,"osd_in_since":1726660457,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+degraded","count":9},{"state_name":"active+recovery_unfound","count":5},{"state_name":"active+recovering+degraded","count":1}],"num_pgs":24,"num_pools":2,"num_objects":30,"data_bytes":17295,"bytes_used":610537472,"bytes_avail":192662990848,"bytes_total":193273528320,"degraded_objects":52,"degraded_total":60,"degraded_ratio":0.8666666666666667,"unfound_objects":27,"unfound_total":30,"unfound_ratio":0.90000000000000002},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-09-18T11:54:11.022915+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:54:23.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': 'c6912087-b15b-4223-a3df-3a06756ee775', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/30 objects unfound (90.000%)', 'count': 27}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 5 pgs recovery_unfound', 'count': 5}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 52/60 objects degraded (86.667%), 10 pgs degraded', 'count': 10}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 63, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 30, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1726660457, 'num_in_osds': 2, 'osd_in_since': 1726660457, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 9}, {'state_name': 'active+recovery_wait+degraded', 'count': 9}, {'state_name': 'active+recovery_unfound', 'count': 5}, {'state_name': 'active+recovering+degraded', 'count': 1}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 30, 'data_bytes': 17295, 'bytes_used': 610537472, 'bytes_avail': 192662990848, 'bytes_total': 193273528320, 'degraded_objects': 52, 'degraded_total': 60, 'degraded_ratio': 0.8666666666666667, 'unfound_objects': 27, 'unfound_total': 30, 'unfound_ratio': 0.9}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 3, 'modified': '2024-09-18T11:54:11.022915+0000', 'services': {'mgr': {'daemons': {'summary': '', 'x': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'mon': {'daemons': {'summary': '', 'a': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'b': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'c': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-18T11:54:23.864 INFO:tasks.rep_lost_unfound_delete.ceph_manager:active! 2024-09-18T11:54:23.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:54:23.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T11:54:23.997 INFO:teuthology.orchestra.run.smithi154.stdout:124554051589 2024-09-18T11:54:23.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:54:24.007 INFO:teuthology.orchestra.run.smithi154.stdout:98784247816 2024-09-18T11:54:24.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:54:24.295 INFO:teuthology.orchestra.run.smithi154.stdout:124554051588 2024-09-18T11:54:24.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 124554051589 got 124554051588 for osd.2 2024-09-18T11:54:24.325 INFO:teuthology.orchestra.run.smithi154.stdout:98784247815 2024-09-18T11:54:24.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 98784247816 got 98784247815 for osd.0 2024-09-18T11:54:25.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:54:25.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:54:25.635 INFO:teuthology.orchestra.run.smithi154.stdout:124554051589 2024-09-18T11:54:25.654 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 124554051589 got 124554051589 for osd.2 2024-09-18T11:54:25.654 DEBUG:teuthology.parallel:result is None 2024-09-18T11:54:25.674 INFO:teuthology.orchestra.run.smithi154.stdout:98784247816 2024-09-18T11:54:25.687 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 98784247816 got 98784247816 for osd.0 2024-09-18T11:54:25.687 DEBUG:teuthology.parallel:result is None 2024-09-18T11:54:25.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:54:26.029 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:26.029 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/30 objects unfound (90.000%)","count":27},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 5 pgs recovery_unfound","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 52/60 objects degraded (86.667%), 10 pgs degraded","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":66,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":30,"num_osds":3,"num_up_osds":2,"osd_up_since":1726660457,"num_in_osds":2,"osd_in_since":1726660457,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+degraded","count":9},{"state_name":"active+recovery_unfound","count":5},{"state_name":"active+recovering+degraded","count":1}],"num_pgs":24,"num_pools":2,"num_objects":30,"data_bytes":17295,"bytes_used":610537472,"bytes_avail":192662990848,"bytes_total":193273528320,"degraded_objects":52,"degraded_total":60,"degraded_ratio":0.8666666666666667,"unfound_objects":27,"unfound_total":30,"unfound_ratio":0.90000000000000002},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-09-18T11:54:11.022915+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:54:26.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': 'c6912087-b15b-4223-a3df-3a06756ee775', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/30 objects unfound (90.000%)', 'count': 27}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 5 pgs recovery_unfound', 'count': 5}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 52/60 objects degraded (86.667%), 10 pgs degraded', 'count': 10}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 66, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 30, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1726660457, 'num_in_osds': 2, 'osd_in_since': 1726660457, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 9}, {'state_name': 'active+recovery_wait+degraded', 'count': 9}, {'state_name': 'active+recovery_unfound', 'count': 5}, {'state_name': 'active+recovering+degraded', 'count': 1}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 30, 'data_bytes': 17295, 'bytes_used': 610537472, 'bytes_avail': 192662990848, 'bytes_total': 193273528320, 'degraded_objects': 52, 'degraded_total': 60, 'degraded_ratio': 0.8666666666666667, 'unfound_objects': 27, 'unfound_total': 30, 'unfound_ratio': 0.9}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 3, 'modified': '2024-09-18T11:54:11.022915+0000', 'services': {'mgr': {'daemons': {'summary': '', 'x': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'mon': {'daemons': {'summary': '', 'a': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'b': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'c': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-18T11:54:26.043 INFO:tasks.rep_lost_unfound_delete:there are 27 unfound objects 2024-09-18T11:54:26.043 DEBUG:teuthology.orchestra.run.smithi154:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.admin -b 4096 -p unfounddel_pool -t 20 bench 240 write' 2024-09-18T11:54:26.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:hints = 1 2024-09-18T11:54:26.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Maintaining 20 concurrent writes of 4096 bytes to objects of size 4096 for up to 240 seconds or 0 objects 2024-09-18T11:54:26.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Object prefix: benchmark_data_smithi154_29013 2024-09-18T11:54:26.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:54:26.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 0 0 0 0 0 0 - 0 2024-09-18T11:54:27.092 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 1 20 2888 2868 11.2025 11.2031 0.00735443 0.00695285 2024-09-18T11:54:28.092 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 2 20 5772 5752 11.2337 11.2656 0.00948009 0.00694444 2024-09-18T11:54:29.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 3 20 8402 8382 10.9133 10.2734 0.0102414 0.00715147 2024-09-18T11:54:30.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 4 20 10922 10902 10.6457 9.84375 0.00947443 0.00732741 2024-09-18T11:54:31.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 5 20 13328 13308 10.3961 9.39844 0.00470327 0.00750908 2024-09-18T11:54:32.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 6 20 15610 15590 10.149 8.91406 0.00854786 0.00768856 2024-09-18T11:54:33.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 7 20 17835 17815 9.94063 8.69141 0.00667036 0.00785102 2024-09-18T11:54:34.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 8 20 19926 19906 9.71897 8.16797 0.00634936 0.00803016 2024-09-18T11:54:35.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 9 20 21983 21963 9.53181 8.03516 0.00710724 0.00818842 2024-09-18T11:54:36.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:36.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 10 20 23946 23926 9.34536 7.66797 0.0205348 0.00835144 2024-09-18T11:54:36.453 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:36.461 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:54:36.474 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-09-18T11:54:35.030992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60502927,"num_objects":14797,"num_object_clones":0,"num_object_copies":29594,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":14797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14824,"num_write_kb":59116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14825,"ondisk_log_size":14825,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":726120,"kb_used_data":105496,"kb_used_omap":8,"kb_used_meta":620599,"kb_avail":188017560,"statfs":{"total":193273528320,"available":192529981440,"internally_reserved":0,"allocated":108027904,"data_stored":107637585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9204,"internal_metadata":635493388},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,10,12,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":60485632,"num_objects":14767,"num_object_clones":0,"num_object_copies":29534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14767,"num_write_kb":59068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004001"},"pg_stats":[{"pgid":"2.f","version":"30'822","reported_seq":864,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:31.289792+0000","last_change":"2024-09-18T11:54:18.444208+0000","last_active":"2024-09-18T11:54:31.289792+0000","last_peered":"2024-09-18T11:54:31.289792+0000","last_clean":"2024-09-18T11:54:31.289792+0000","last_became_active":"2024-09-18T11:54:18.443917+0000","last_became_peered":"2024-09-18T11:54:18.443917+0000","last_unstale":"2024-09-18T11:54:31.289792+0000","last_undegraded":"2024-09-18T11:54:31.289792+0000","last_fullsized":"2024-09-18T11:54:31.289792+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:52:17.496947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3366912,"num_objects":822,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":822,"num_write_kb":3288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'1020","reported_seq":1037,"reported_epoch":30,"state":"active+recovery_unfound+degraded","last_fresh":"2024-09-18T11:54:32.409725+0000","last_change":"2024-09-18T11:54:26.162268+0000","last_active":"2024-09-18T11:54:32.409725+0000","last_peered":"2024-09-18T11:54:32.409725+0000","last_clean":"2024-09-18T11:53:53.108530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:54:32.409725+0000","last_undegraded":"2024-09-18T11:54:18.446754+0000","last_fullsized":"2024-09-18T11:54:32.409725+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1020,"log_dups_size":0,"ondisk_log_size":1020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:31:16.015386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4162770,"num_objects":1018,"num_object_clones":0,"num_object_copies":2036,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1020,"num_write_kb":4068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1018}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"30'1026","reported_seq":1038,"reported_epoch":30,"state":"active+recovery_unfound+degraded","last_fresh":"2024-09-18T11:54:32.415216+0000","last_change":"2024-09-18T11:54:26.131005+0000","last_active":"2024-09-18T11:54:32.415216+0000","last_peered":"2024-09-18T11:54:32.415216+0000","last_clean":"2024-09-18T11:53:53.677796+0000","last_became_active":"2024-09-18T11:54:18.439908+0000","last_became_peered":"2024-09-18T11:54:18.439908+0000","last_unstale":"2024-09-18T11:54:32.415216+0000","last_undegraded":"2024-09-18T11:54:18.445446+0000","last_fullsized":"2024-09-18T11:54:32.415216+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1026,"log_dups_size":0,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:28:09.678683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4190825,"num_objects":1024,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1026,"num_write_kb":4094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1024}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"30'1012","reported_seq":1032,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:32.417934+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:54:32.417934+0000","last_peered":"2024-09-18T11:54:32.417934+0000","last_clean":"2024-09-18T11:54:32.417934+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:54:32.417934+0000","last_undegraded":"2024-09-18T11:54:32.417934+0000","last_fullsized":"2024-09-18T11:54:32.417934+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:35.240042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4145152,"num_objects":1012,"num_object_clones":0,"num_object_copies":2024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1012,"num_write_kb":4048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"30'836","reported_seq":886,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:31.306006+0000","last_change":"2024-09-18T11:54:18.443818+0000","last_active":"2024-09-18T11:54:31.306006+0000","last_peered":"2024-09-18T11:54:31.306006+0000","last_clean":"2024-09-18T11:53:53.013443+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:54:31.306006+0000","last_undegraded":"2024-09-18T11:54:18.431175+0000","last_fullsized":"2024-09-18T11:54:31.306006+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":836,"log_dups_size":0,"ondisk_log_size":836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:47:11.063973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3405010,"num_objects":833,"num_object_clones":0,"num_object_copies":1666,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":836,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":833}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"30'899","reported_seq":942,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:31.305485+0000","last_change":"2024-09-18T11:54:18.442338+0000","last_active":"2024-09-18T11:54:31.305485+0000","last_peered":"2024-09-18T11:54:31.305485+0000","last_clean":"2024-09-18T11:53:53.869066+0000","last_became_active":"2024-09-18T11:54:18.441285+0000","last_became_peered":"2024-09-18T11:54:18.441285+0000","last_unstale":"2024-09-18T11:54:31.305485+0000","last_undegraded":"2024-09-18T11:54:18.433470+0000","last_fullsized":"2024-09-18T11:54:31.305485+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":899,"log_dups_size":0,"ondisk_log_size":899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:50:27.827107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3671250,"num_objects":898,"num_object_clones":0,"num_object_copies":1796,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":899,"num_write_kb":3587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":898}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"30'829","reported_seq":874,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:31.299505+0000","last_change":"2024-09-18T11:54:18.446054+0000","last_active":"2024-09-18T11:54:31.299505+0000","last_peered":"2024-09-18T11:54:31.299505+0000","last_clean":"2024-09-18T11:53:53.251624+0000","last_became_active":"2024-09-18T11:54:18.445771+0000","last_became_peered":"2024-09-18T11:54:18.445771+0000","last_unstale":"2024-09-18T11:54:31.299505+0000","last_undegraded":"2024-09-18T11:54:18.434522+0000","last_fullsized":"2024-09-18T11:54:31.299505+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":829,"log_dups_size":0,"ondisk_log_size":829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:54:48.333477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3380434,"num_objects":827,"num_object_clones":0,"num_object_copies":1654,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":829,"num_write_kb":3304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":827}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"30'899","reported_seq":950,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:31.308107+0000","last_change":"2024-09-18T11:54:18.446046+0000","last_active":"2024-09-18T11:54:31.308107+0000","last_peered":"2024-09-18T11:54:31.308107+0000","last_clean":"2024-09-18T11:53:43.208142+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:54:31.308107+0000","last_undegraded":"2024-09-18T11:54:18.434814+0000","last_fullsized":"2024-09-18T11:54:31.308107+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":899,"log_dups_size":0,"ondisk_log_size":899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:22.388387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3678825,"num_objects":899,"num_object_clones":0,"num_object_copies":1798,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":899,"num_write_kb":3593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":899}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441722+0000","last_change":"2024-09-18T11:54:18.441722+0000","last_active":"2024-09-18T11:54:18.441722+0000","last_peered":"2024-09-18T11:54:18.441722+0000","last_clean":"2024-09-18T11:54:18.441722+0000","last_became_active":"2024-09-18T11:54:18.440263+0000","last_became_peered":"2024-09-18T11:54:18.440263+0000","last_unstale":"2024-09-18T11:54:18.441722+0000","last_undegraded":"2024-09-18T11:54:18.441722+0000","last_fullsized":"2024-09-18T11:54:18.441722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:27:26.196691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"30'1030","reported_seq":1040,"reported_epoch":30,"state":"active+recovery_unfound+degraded","last_fresh":"2024-09-18T11:54:32.415183+0000","last_change":"2024-09-18T11:54:26.150267+0000","last_active":"2024-09-18T11:54:32.415183+0000","last_peered":"2024-09-18T11:54:32.415183+0000","last_clean":"2024-09-18T11:53:53.726368+0000","last_became_active":"2024-09-18T11:54:18.440195+0000","last_became_peered":"2024-09-18T11:54:18.440195+0000","last_unstale":"2024-09-18T11:54:32.415183+0000","last_undegraded":"2024-09-18T11:54:18.447311+0000","last_fullsized":"2024-09-18T11:54:32.415183+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1030,"log_dups_size":0,"ondisk_log_size":1030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:11:21.352233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4199634,"num_objects":1027,"num_object_clones":0,"num_object_copies":2054,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1030,"num_write_kb":4104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1027}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439949+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:54:18.439949+0000","last_peered":"2024-09-18T11:54:18.439949+0000","last_clean":"2024-09-18T11:54:18.439949+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:54:18.439949+0000","last_undegraded":"2024-09-18T11:54:18.439949+0000","last_fullsized":"2024-09-18T11:54:18.439949+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:14:25.540910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"30'882","reported_seq":921,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:31.292806+0000","last_change":"2024-09-18T11:54:18.445976+0000","last_active":"2024-09-18T11:54:31.292806+0000","last_peered":"2024-09-18T11:54:31.292806+0000","last_clean":"2024-09-18T11:53:53.913371+0000","last_became_active":"2024-09-18T11:54:18.445692+0000","last_became_peered":"2024-09-18T11:54:18.445692+0000","last_unstale":"2024-09-18T11:54:31.292806+0000","last_undegraded":"2024-09-18T11:54:18.433038+0000","last_fullsized":"2024-09-18T11:54:31.292806+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:45:22.401658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3589330,"num_objects":878,"num_object_clones":0,"num_object_copies":1756,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":882,"num_write_kb":3508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":878}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":51,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.441089+0000","last_change":"2024-09-18T11:54:18.441264+0000","last_active":"2024-09-18T11:54:18.441089+0000","last_peered":"2024-09-18T11:54:18.441089+0000","last_clean":"2024-09-18T11:54:03.316651+0000","last_became_active":"2024-09-18T11:54:18.440963+0000","last_became_peered":"2024-09-18T11:54:18.440963+0000","last_unstale":"2024-09-18T11:54:18.441089+0000","last_undegraded":"2024-09-18T11:54:18.432164+0000","last_fullsized":"2024-09-18T11:54:18.441089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:23:22.104360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"30'1020","reported_seq":1038,"reported_epoch":30,"state":"active+recovery_unfound+degraded","last_fresh":"2024-09-18T11:54:32.406281+0000","last_change":"2024-09-18T11:54:26.128178+0000","last_active":"2024-09-18T11:54:32.406281+0000","last_peered":"2024-09-18T11:54:32.406281+0000","last_clean":"2024-09-18T11:53:52.962496+0000","last_became_active":"2024-09-18T11:54:18.438876+0000","last_became_peered":"2024-09-18T11:54:18.438876+0000","last_unstale":"2024-09-18T11:54:32.406281+0000","last_undegraded":"2024-09-18T11:54:18.447897+0000","last_fullsized":"2024-09-18T11:54:32.406281+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1020,"log_dups_size":0,"ondisk_log_size":1020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:57:40.116608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4166866,"num_objects":1019,"num_object_clones":0,"num_object_copies":2038,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1020,"num_write_kb":4071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1019}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439859+0000","last_change":"2024-09-18T11:54:18.439859+0000","last_active":"2024-09-18T11:54:18.439859+0000","last_peered":"2024-09-18T11:54:18.439859+0000","last_clean":"2024-09-18T11:54:18.439859+0000","last_became_active":"2024-09-18T11:54:18.438620+0000","last_became_peered":"2024-09-18T11:54:18.438620+0000","last_unstale":"2024-09-18T11:54:18.439859+0000","last_undegraded":"2024-09-18T11:54:18.439859+0000","last_fullsized":"2024-09-18T11:54:18.439859+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:09:25.244649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"30'1026","reported_seq":1043,"reported_epoch":30,"state":"active+recovery_unfound+degraded","last_fresh":"2024-09-18T11:54:32.398084+0000","last_change":"2024-09-18T11:54:26.127714+0000","last_active":"2024-09-18T11:54:32.398084+0000","last_peered":"2024-09-18T11:54:32.398084+0000","last_clean":"2024-09-18T11:53:53.346427+0000","last_became_active":"2024-09-18T11:54:18.441199+0000","last_became_peered":"2024-09-18T11:54:18.441199+0000","last_unstale":"2024-09-18T11:54:32.398084+0000","last_undegraded":"2024-09-18T11:54:18.448486+0000","last_fullsized":"2024-09-18T11:54:32.398084+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":1026,"log_dups_size":0,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:56:18.400692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4187346,"num_objects":1024,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1026,"num_write_kb":4091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1024}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"30'894","reported_seq":938,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:31.289679+0000","last_change":"2024-09-18T11:54:18.445855+0000","last_active":"2024-09-18T11:54:31.289679+0000","last_peered":"2024-09-18T11:54:31.289679+0000","last_clean":"2024-09-18T11:53:46.243258+0000","last_became_active":"2024-09-18T11:54:18.445615+0000","last_became_peered":"2024-09-18T11:54:18.445615+0000","last_unstale":"2024-09-18T11:54:31.289679+0000","last_undegraded":"2024-09-18T11:54:18.433012+0000","last_fullsized":"2024-09-18T11:54:31.289679+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:33:38.875122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3658345,"num_objects":894,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":894,"num_write_kb":3573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.442188+0000","last_change":"2024-09-18T11:54:18.442188+0000","last_active":"2024-09-18T11:54:18.442188+0000","last_peered":"2024-09-18T11:54:18.442188+0000","last_clean":"2024-09-18T11:54:18.442188+0000","last_became_active":"2024-09-18T11:54:18.441831+0000","last_became_peered":"2024-09-18T11:54:18.441831+0000","last_unstale":"2024-09-18T11:54:18.442188+0000","last_undegraded":"2024-09-18T11:54:18.442188+0000","last_fullsized":"2024-09-18T11:54:18.442188+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:20:27.929651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"30'867","reported_seq":912,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:31.298393+0000","last_change":"2024-09-18T11:54:18.441937+0000","last_active":"2024-09-18T11:54:31.298393+0000","last_peered":"2024-09-18T11:54:31.298393+0000","last_clean":"2024-09-18T11:53:53.823053+0000","last_became_active":"2024-09-18T11:54:18.441659+0000","last_became_peered":"2024-09-18T11:54:18.441659+0000","last_unstale":"2024-09-18T11:54:31.298393+0000","last_undegraded":"2024-09-18T11:54:18.432961+0000","last_fullsized":"2024-09-18T11:54:31.298393+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:12:36.428174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3532603,"num_objects":865,"num_object_clones":0,"num_object_copies":1730,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":867,"num_write_kb":3452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":865}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440580+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:54:18.440580+0000","last_peered":"2024-09-18T11:54:18.440580+0000","last_clean":"2024-09-18T11:54:18.440580+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:54:18.440580+0000","last_undegraded":"2024-09-18T11:54:18.440580+0000","last_fullsized":"2024-09-18T11:54:18.440580+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:10:36.497525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"30'873","reported_seq":918,"reported_epoch":30,"state":"active+recovering","last_fresh":"2024-09-18T11:54:31.290270+0000","last_change":"2024-09-18T11:54:26.134693+0000","last_active":"2024-09-18T11:54:31.290270+0000","last_peered":"2024-09-18T11:54:31.290270+0000","last_clean":"2024-09-18T11:53:46.242858+0000","last_became_active":"2024-09-18T11:54:18.440831+0000","last_became_peered":"2024-09-18T11:54:18.440831+0000","last_unstale":"2024-09-18T11:54:31.290270+0000","last_undegraded":"2024-09-18T11:54:31.290270+0000","last_fullsized":"2024-09-18T11:54:31.290270+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":873,"log_dups_size":0,"ondisk_log_size":873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:18:26.703672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3568850,"num_objects":873,"num_object_clones":0,"num_object_copies":1746,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":873,"num_write_kb":3486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":873}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441579+0000","last_change":"2024-09-18T11:54:18.441579+0000","last_active":"2024-09-18T11:54:18.441579+0000","last_peered":"2024-09-18T11:54:18.441579+0000","last_clean":"2024-09-18T11:54:18.441579+0000","last_became_active":"2024-09-18T11:54:18.440490+0000","last_became_peered":"2024-09-18T11:54:18.440490+0000","last_unstale":"2024-09-18T11:54:18.441579+0000","last_undegraded":"2024-09-18T11:54:18.441579+0000","last_fullsized":"2024-09-18T11:54:18.441579+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:04.506685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440411+0000","last_change":"2024-09-18T11:54:18.440411+0000","last_active":"2024-09-18T11:54:18.440411+0000","last_peered":"2024-09-18T11:54:18.440411+0000","last_clean":"2024-09-18T11:54:18.440411+0000","last_became_active":"2024-09-18T11:54:18.439759+0000","last_became_peered":"2024-09-18T11:54:18.439759+0000","last_unstale":"2024-09-18T11:54:18.440411+0000","last_undegraded":"2024-09-18T11:54:18.440411+0000","last_fullsized":"2024-09-18T11:54:18.440411+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:02.939140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"30'887","reported_seq":932,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:31.302303+0000","last_change":"2024-09-18T11:54:18.441905+0000","last_active":"2024-09-18T11:54:31.302303+0000","last_peered":"2024-09-18T11:54:31.302303+0000","last_clean":"2024-09-18T11:53:53.582527+0000","last_became_active":"2024-09-18T11:54:18.440844+0000","last_became_peered":"2024-09-18T11:54:18.440844+0000","last_unstale":"2024-09-18T11:54:31.302303+0000","last_undegraded":"2024-09-18T11:54:18.431783+0000","last_fullsized":"2024-09-18T11:54:31.302303+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":887,"log_dups_size":0,"ondisk_log_size":887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:38:24.322706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3598756,"num_objects":882,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":887,"num_write_kb":3519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":881}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":60502908,"num_objects":14795,"num_object_clones":0,"num_object_copies":29590,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":51,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":14795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14822,"num_write_kb":59114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":123842560,"data_stored":123710358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14822,"ondisk_log_size":14822,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051591,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368340,"kb_used_data":57996,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94003500,"statfs":{"total":96636764160,"available":96259584000,"internally_reserved":0,"allocated":59387904,"data_stored":59212178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,6,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247818,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":357780,"kb_used_data":47500,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94014060,"statfs":{"total":96636764160,"available":96270397440,"internally_reserved":0,"allocated":48640000,"data_stored":48425407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,4,10,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56627200,"data_stored":56592410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67100672,"data_stored":67100672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:36.475 INFO:tasks.rep_lost_unfound_delete:no unfound in 2.f 2024-09-18T11:54:36.475 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.e state active+recovery_unfound+degraded 2024-09-18T11:54:36.475 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.e list_unfound '{}' 2024-09-18T11:54:36.640 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:36.641 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:36.641 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:36.641 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:36.641 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:36.641 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:36.641 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:36.641 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:36.641 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:36.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:36.641 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 2, 2024-09-18T11:54:36.641 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 2, 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existing_4", 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 1224592654, 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'3", 2024-09-18T11:54:36.642 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "19'1", 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existing_6", 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 3101443614, 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:36.643 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'4", 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "19'2", 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "NotRecovering", 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:36.644 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:36.645 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:36.645 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:36.645 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:36.645 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:36.645 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:36.661 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.e on osd.2 2024-09-18T11:54:36.661 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.e mark_unfound_lost delete 2024-09-18T11:54:36.842 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:54:36.854 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.d state active+recovery_unfound+degraded 2024-09-18T11:54:36.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.d list_unfound '{}' 2024-09-18T11:54:37.050 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:37.051 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:37.051 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:37.051 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:37.051 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:37.051 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:37.051 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:37.051 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:37.051 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:37.051 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:37.051 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 1, 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 1, 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existed_7", 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 2358579165, 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:37.052 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'3", 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "NotRecovering", 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:37.053 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:37.054 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:37.054 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:37.072 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.d on osd.2 2024-09-18T11:54:37.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.d mark_unfound_lost delete 2024-09-18T11:54:37.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 11 20 25532 25512 9.05894 6.19531 0.0116771 0.00861692 2024-09-18T11:54:37.279 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-18T11:54:37.281 INFO:tasks.rep_lost_unfound_delete:no unfound in 2.c 2024-09-18T11:54:37.281 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.b state active+recovery_wait+degraded 2024-09-18T11:54:37.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.b list_unfound '{}' 2024-09-18T11:54:37.480 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 2, 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 2, 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existed_3", 2024-09-18T11:54:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 943960171, 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'5", 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existing_1", 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 2885884443, 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'4", 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "19'1", 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-09-18T11:54:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "Active", 2024-09-18T11:54:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:37.484 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:37.493 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.b on osd.0 2024-09-18T11:54:37.493 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.b mark_unfound_lost delete 2024-09-18T11:54:37.673 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:54:37.686 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.a state active+recovery_wait+degraded 2024-09-18T11:54:37.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.a list_unfound '{}' 2024-09-18T11:54:37.862 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:37.862 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:37.862 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 2, 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 2, 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existing_9", 2024-09-18T11:54:37.863 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 1217469130, 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'3", 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "19'1", 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:37.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "new_9", 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 170713514, 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'2", 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:37.865 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "Active", 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:37.866 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:37.878 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.a on osd.0 2024-09-18T11:54:37.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.a mark_unfound_lost delete 2024-09-18T11:54:38.061 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:54:38.073 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.9 state active+recovery_wait+degraded 2024-09-18T11:54:38.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.9 list_unfound '{}' 2024-09-18T11:54:38.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 12 20 27168 27148 8.83652 6.39062 0.0099042 0.00883345 2024-09-18T11:54:38.232 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:38.232 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:38.232 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:38.232 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:38.232 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:38.232 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 2, 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 2, 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existing_5", 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 2998549577, 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:38.233 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'4", 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "19'2", 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existing_2", 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 4032093161, 2024-09-18T11:54:38.234 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'3", 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "19'1", 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "Active", 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:38.235 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:38.236 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:38.236 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:38.236 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:38.236 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:38.252 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.9 on osd.0 2024-09-18T11:54:38.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.9 mark_unfound_lost delete 2024-09-18T11:54:38.457 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:54:38.471 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.8 state active+recovery_wait+degraded 2024-09-18T11:54:38.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.8 list_unfound '{}' 2024-09-18T11:54:38.637 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:38.637 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:38.637 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:38.637 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:38.637 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:38.637 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:38.637 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 1, 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 1, 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "new_8", 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 2730327960, 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:38.638 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'1", 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "Active", 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:38.639 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:38.640 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:38.640 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:38.640 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:38.649 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.8 on osd.0 2024-09-18T11:54:38.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.8 mark_unfound_lost delete 2024-09-18T11:54:38.831 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-18T11:54:38.847 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.0 2024-09-18T11:54:38.847 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.3 state active+recovery_unfound+degraded 2024-09-18T11:54:38.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.3 list_unfound '{}' 2024-09-18T11:54:39.001 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:39.001 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:39.001 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:39.001 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:39.001 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:39.001 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:39.001 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:39.001 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 2, 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 2, 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existing_8", 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 1901787523, 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'5", 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "19'3", 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existed_1", 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:39.003 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 3122379363, 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'4", 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "NotRecovering", 2024-09-18T11:54:39.004 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:39.005 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:39.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:39.005 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:39.005 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:39.005 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:39.005 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:39.005 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:39.005 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:39.015 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.3 on osd.2 2024-09-18T11:54:39.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.3 mark_unfound_lost delete 2024-09-18T11:54:39.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 13 20 28757 28737 8.63421 6.20703 0.0108727 0.00903937 2024-09-18T11:54:39.201 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:54:39.213 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.1 2024-09-18T11:54:39.213 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.2 state active+recovery_wait+degraded 2024-09-18T11:54:39.213 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.2 list_unfound '{}' 2024-09-18T11:54:39.390 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:39.390 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:39.390 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 2, 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 2, 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:39.391 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existed_6", 2024-09-18T11:54:39.392 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:39.392 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:39.392 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 1708673474, 2024-09-18T11:54:39.392 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:39.392 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:39.392 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:39.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:39.392 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'5", 2024-09-18T11:54:39.392 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:39.392 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:39.392 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:39.392 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existed_9", 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 831402034, 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'6", 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:39.393 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "Active", 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:39.394 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:39.395 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:39.402 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.2 on osd.0 2024-09-18T11:54:39.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.2 mark_unfound_lost delete 2024-09-18T11:54:39.591 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:54:39.595 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.2 2024-09-18T11:54:39.595 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.1 state active+recovery_unfound+degraded 2024-09-18T11:54:39.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.1 list_unfound '{}' 2024-09-18T11:54:39.613 INFO:tasks.ceph.mon.a.smithi154.stderr:2024-09-18T11:54:39.608+0000 7f2c804bf700 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 3 pgs recovery_unfound (PG_DAMAGED) 2024-09-18T11:54:39.758 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:39.759 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:39.759 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:39.759 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:39.759 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:39.759 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:39.759 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:39.759 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:39.759 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:39.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:39.759 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 2, 2024-09-18T11:54:39.759 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 2, 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existing_3", 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 459768081, 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'3", 2024-09-18T11:54:39.760 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "19'1", 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "new_3", 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 480221105, 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:39.761 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'2", 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "NotRecovering", 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:39.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:39.763 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:39.763 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:39.763 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:39.763 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:39.763 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:39.776 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.1 on osd.2 2024-09-18T11:54:39.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.1 mark_unfound_lost delete 2024-09-18T11:54:40.005 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:54:40.017 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.3 2024-09-18T11:54:40.017 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.0 state active+recovery_unfound+degraded 2024-09-18T11:54:40.018 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.0 list_unfound '{}' 2024-09-18T11:54:40.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 14 19 30344 30325 8.46052 6.20312 0.0132156 0.00922614 2024-09-18T11:54:40.190 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:40.190 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:40.190 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:40.190 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:40.191 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:40.191 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:40.191 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:40.191 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:40.191 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:40.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:40.191 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 2, 2024-09-18T11:54:40.191 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 2, 2024-09-18T11:54:40.191 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:40.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:40.192 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:40.192 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "new_5", 2024-09-18T11:54:40.192 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:40.192 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:40.192 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 4248945696, 2024-09-18T11:54:40.192 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:40.192 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:40.192 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:40.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:40.192 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'3", 2024-09-18T11:54:40.193 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:40.193 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:40.193 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:40.193 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:40.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:40.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:40.193 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:40.193 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existed_5", 2024-09-18T11:54:40.193 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:40.193 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:40.193 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 4212789968, 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'4", 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "NotRecovering", 2024-09-18T11:54:40.194 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:40.195 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:40.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:40.195 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:40.195 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:40.195 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:40.195 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:40.195 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:40.195 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:40.211 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.0 on osd.2 2024-09-18T11:54:40.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.0 mark_unfound_lost delete 2024-09-18T11:54:40.416 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:54:40.429 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.7 state active+recovery_wait+degraded 2024-09-18T11:54:40.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.7 list_unfound '{}' 2024-09-18T11:54:40.602 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:40.602 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:40.602 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:40.602 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:40.603 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:40.603 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:40.603 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:40.603 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:40.603 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:40.603 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:40.603 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 1, 2024-09-18T11:54:40.603 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 1, 2024-09-18T11:54:40.603 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:40.603 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:40.603 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:40.604 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "new_7", 2024-09-18T11:54:40.604 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:40.604 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:40.604 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 3761017527, 2024-09-18T11:54:40.604 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:40.604 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:40.604 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:40.604 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:40.604 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'1", 2024-09-18T11:54:40.604 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:40.604 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:40.605 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:40.605 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:40.605 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:40.605 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:40.605 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "Active", 2024-09-18T11:54:40.605 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:40.605 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:40.605 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:40.605 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:40.605 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:40.605 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:40.605 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:40.606 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:40.606 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:40.618 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.7 on osd.0 2024-09-18T11:54:40.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.7 mark_unfound_lost delete 2024-09-18T11:54:40.831 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-18T11:54:40.846 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.4 2024-09-18T11:54:40.847 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.6 state active+recovery_wait+degraded 2024-09-18T11:54:40.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.6 list_unfound '{}' 2024-09-18T11:54:41.003 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:41.003 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:41.003 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:41.003 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 3, 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 3, 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existed_8", 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:41.004 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 1861424966, 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'5", 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "new_6", 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 3910131350, 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'4", 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "new_4", 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 2787839990, 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'3", 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:41.007 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:41.008 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "Active", 2024-09-18T11:54:41.008 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:41.008 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:41.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:41.008 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:41.008 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:41.008 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:41.008 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:41.008 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:41.008 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:41.018 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.6 on osd.0 2024-09-18T11:54:41.018 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.6 mark_unfound_lost delete 2024-09-18T11:54:41.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 15 20 31891 31871 8.29906 6.03906 0.0119768 0.00940776 2024-09-18T11:54:41.216 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-18T11:54:41.228 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.5 2024-09-18T11:54:41.228 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.5 state active+recovering 2024-09-18T11:54:41.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.5 list_unfound '{}' 2024-09-18T11:54:41.409 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:41.410 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:41.410 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:41.410 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:41.410 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:41.410 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:41.410 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:41.410 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:41.410 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:41.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.410 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 2, 2024-09-18T11:54:41.410 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 2, 2024-09-18T11:54:41.410 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "new_1", 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 26281365, 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'1", 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:41.411 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:41.412 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:41.412 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:41.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:41.412 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:41.412 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "new_2", 2024-09-18T11:54:41.412 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:41.412 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:41.412 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 3196340981, 2024-09-18T11:54:41.412 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:41.412 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:41.412 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'2", 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "Active", 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:41.413 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:41.414 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:41.414 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:41.414 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:41.414 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:41.424 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.5 on osd.0 2024-09-18T11:54:41.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.5 mark_unfound_lost delete 2024-09-18T11:54:41.617 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-18T11:54:41.629 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.6 2024-09-18T11:54:41.629 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.7 2024-09-18T11:54:41.629 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.4 state active+recovery_wait+degraded 2024-09-18T11:54:41.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.4 list_unfound '{}' 2024-09-18T11:54:41.807 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:54:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "offset": { 2024-09-18T11:54:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "", 2024-09-18T11:54:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": 0, 2024-09-18T11:54:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 0, 2024-09-18T11:54:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": -9223372036854775808, 2024-09-18T11:54:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "num_missing": 4, 2024-09-18T11:54:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "num_unfound": 3, 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "objects": [ 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existed_4", 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 873891716, 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'8", 2024-09-18T11:54:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existing_7", 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 3996155460, 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'9", 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "19'6", 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0", 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": { 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "oid": "existed_2", 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "snapid": -2, 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "hash": 1767667508, 2024-09-18T11:54:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 0, 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "pool": 2, 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "namespace": "" 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "need": "21'7", 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "have": "0'0", 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "flags": "none", 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "locations": [] 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "state": "Active", 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "available_might_have_unfound": true, 2024-09-18T11:54:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "might_have_unfound": [ 2024-09-18T11:54:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-18T11:54:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": "1", 2024-09-18T11:54:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: "status": "osd is down" 2024-09-18T11:54:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-18T11:54:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-18T11:54:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: "more": false 2024-09-18T11:54:41.813 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:54:41.827 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.4 on osd.0 2024-09-18T11:54:41.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.4 mark_unfound_lost delete 2024-09-18T11:54:42.015 INFO:teuthology.orchestra.run.smithi154.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-18T11:54:42.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 5 2024-09-18T11:54:42.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 16 19 33368 33349 8.14118 5.77344 0.0134617 0.00959034 2024-09-18T11:54:42.184 INFO:teuthology.orchestra.run.smithi154.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-18T11:54:42.205 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 5 2024-09-18T11:54:42.372 INFO:teuthology.orchestra.run.smithi154.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-18T11:54:42.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T11:54:42.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T11:54:42.576 INFO:teuthology.orchestra.run.smithi154.stdout:98784247821 2024-09-18T11:54:42.576 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:54:42.605 INFO:teuthology.orchestra.run.smithi154.stdout:124554051594 2024-09-18T11:54:42.606 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:54:43.021 INFO:teuthology.orchestra.run.smithi154.stdout:98784247819 2024-09-18T11:54:43.059 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 98784247821 got 98784247819 for osd.0 2024-09-18T11:54:43.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 17 20 34687 34667 7.96511 5.14844 0.0149642 0.00980013 2024-09-18T11:54:43.127 INFO:teuthology.orchestra.run.smithi154.stdout:124554051592 2024-09-18T11:54:43.146 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 124554051594 got 124554051592 for osd.2 2024-09-18T11:54:44.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T11:54:44.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 18 20 36295 36275 7.87153 6.28125 0.0134926 0.00991755 2024-09-18T11:54:44.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T11:54:44.549 INFO:teuthology.orchestra.run.smithi154.stdout:98784247821 2024-09-18T11:54:44.580 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 98784247821 got 98784247821 for osd.0 2024-09-18T11:54:44.580 DEBUG:teuthology.parallel:result is None 2024-09-18T11:54:44.651 INFO:teuthology.orchestra.run.smithi154.stdout:124554051594 2024-09-18T11:54:44.672 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 124554051594 got 124554051594 for osd.2 2024-09-18T11:54:44.673 DEBUG:teuthology.parallel:result is None 2024-09-18T11:54:44.673 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for recovery to complete 2024-09-18T11:54:44.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:45.066 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:45.074 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:54:45.088 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-18T11:54:45.034145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":139215759,"num_objects":34014,"num_object_clones":0,"num_object_copies":68028,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34041,"num_write_kb":135984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34069,"ondisk_log_size":34069,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":861804,"kb_used_data":266140,"kb_used_omap":8,"kb_used_meta":595639,"kb_avail":187881876,"statfs":{"total":193273528320,"available":192391041024,"internally_reserved":0,"allocated":272527360,"data_stored":272150957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9204,"internal_metadata":609934348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,4,9,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":78712832,"num_objects":19217,"num_object_clones":0,"num_object_copies":38434,"num_objects_missing_on_primary":-9,"num_objects_missing":-9,"num_objects_degraded":-18,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":19217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19217,"num_write_kb":76868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003704"},"pg_stats":[{"pgid":"2.f","version":"30'2103","reported_seq":2145,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.545723+0000","last_change":"2024-09-18T11:54:18.444208+0000","last_active":"2024-09-18T11:54:42.545723+0000","last_peered":"2024-09-18T11:54:42.545723+0000","last_clean":"2024-09-18T11:54:42.545723+0000","last_became_active":"2024-09-18T11:54:18.443917+0000","last_became_peered":"2024-09-18T11:54:18.443917+0000","last_unstale":"2024-09-18T11:54:42.545723+0000","last_undegraded":"2024-09-18T11:54:42.545723+0000","last_fullsized":"2024-09-18T11:54:42.545723+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:52:17.496947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8613888,"num_objects":2103,"num_object_clones":0,"num_object_copies":4206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2103,"num_write_kb":8412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'2113","reported_seq":2140,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.563932+0000","last_change":"2024-09-18T11:54:36.889194+0000","last_active":"2024-09-18T11:54:42.563932+0000","last_peered":"2024-09-18T11:54:42.563932+0000","last_clean":"2024-09-18T11:54:42.563932+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:54:42.563932+0000","last_undegraded":"2024-09-18T11:54:42.563932+0000","last_fullsized":"2024-09-18T11:54:42.563932+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8631506,"num_objects":2109,"num_object_clones":0,"num_object_copies":4218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2111,"num_write_kb":8432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"30'2068","reported_seq":2087,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.561204+0000","last_change":"2024-09-18T11:54:37.294288+0000","last_active":"2024-09-18T11:54:42.561204+0000","last_peered":"2024-09-18T11:54:42.561204+0000","last_clean":"2024-09-18T11:54:42.561204+0000","last_became_active":"2024-09-18T11:54:18.439908+0000","last_became_peered":"2024-09-18T11:54:18.439908+0000","last_unstale":"2024-09-18T11:54:42.561204+0000","last_undegraded":"2024-09-18T11:54:42.561204+0000","last_fullsized":"2024-09-18T11:54:42.561204+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2068,"log_dups_size":0,"ondisk_log_size":2068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454761,"num_objects":2065,"num_object_clones":0,"num_object_copies":4130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2067,"num_write_kb":8258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"30'2145","reported_seq":2165,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.566024+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:54:42.566024+0000","last_peered":"2024-09-18T11:54:42.566024+0000","last_clean":"2024-09-18T11:54:42.566024+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:54:42.566024+0000","last_undegraded":"2024-09-18T11:54:42.566024+0000","last_fullsized":"2024-09-18T11:54:42.566024+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2145,"log_dups_size":0,"ondisk_log_size":2145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:35.240042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8785920,"num_objects":2145,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":8580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"30'2088","reported_seq":2137,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.545115+0000","last_change":"2024-09-18T11:54:18.443818+0000","last_active":"2024-09-18T11:54:42.545115+0000","last_peered":"2024-09-18T11:54:42.545115+0000","last_clean":"2024-09-18T11:53:53.013443+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:54:42.545115+0000","last_undegraded":"2024-09-18T11:54:18.431175+0000","last_fullsized":"2024-09-18T11:54:42.545115+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8525010,"num_objects":2083,"num_object_clones":0,"num_object_copies":4166,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":8328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2083}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"30'2129","reported_seq":2173,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.539875+0000","last_change":"2024-09-18T11:54:18.442338+0000","last_active":"2024-09-18T11:54:42.539875+0000","last_peered":"2024-09-18T11:54:42.539875+0000","last_clean":"2024-09-18T11:53:53.869066+0000","last_became_active":"2024-09-18T11:54:18.441285+0000","last_became_peered":"2024-09-18T11:54:18.441285+0000","last_unstale":"2024-09-18T11:54:42.539875+0000","last_undegraded":"2024-09-18T11:54:18.433470+0000","last_fullsized":"2024-09-18T11:54:42.539875+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2129,"log_dups_size":0,"ondisk_log_size":2129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8701138,"num_objects":2126,"num_object_clones":0,"num_object_copies":4252,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2127,"num_write_kb":8499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2126}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"30'2080","reported_seq":2124,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.506365+0000","last_change":"2024-09-18T11:54:18.446054+0000","last_active":"2024-09-18T11:54:42.506365+0000","last_peered":"2024-09-18T11:54:42.506365+0000","last_clean":"2024-09-18T11:53:53.251624+0000","last_became_active":"2024-09-18T11:54:18.445771+0000","last_became_peered":"2024-09-18T11:54:18.445771+0000","last_unstale":"2024-09-18T11:54:42.506365+0000","last_undegraded":"2024-09-18T11:54:18.434522+0000","last_fullsized":"2024-09-18T11:54:42.506365+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2080,"log_dups_size":0,"ondisk_log_size":2080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8496338,"num_objects":2076,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2078,"num_write_kb":8300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2076}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"30'2209","reported_seq":2261,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.522184+0000","last_change":"2024-09-18T11:54:18.446046+0000","last_active":"2024-09-18T11:54:42.522184+0000","last_peered":"2024-09-18T11:54:42.522184+0000","last_clean":"2024-09-18T11:53:43.208142+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:54:42.522184+0000","last_undegraded":"2024-09-18T11:54:18.434814+0000","last_fullsized":"2024-09-18T11:54:42.522184+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2209,"log_dups_size":0,"ondisk_log_size":2209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9040489,"num_objects":2208,"num_object_clones":0,"num_object_copies":4416,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":8829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2208}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441722+0000","last_change":"2024-09-18T11:54:18.441722+0000","last_active":"2024-09-18T11:54:18.441722+0000","last_peered":"2024-09-18T11:54:18.441722+0000","last_clean":"2024-09-18T11:54:18.441722+0000","last_became_active":"2024-09-18T11:54:18.440263+0000","last_became_peered":"2024-09-18T11:54:18.440263+0000","last_unstale":"2024-09-18T11:54:18.441722+0000","last_undegraded":"2024-09-18T11:54:18.441722+0000","last_fullsized":"2024-09-18T11:54:18.441722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:27:26.196691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"30'2111","reported_seq":2131,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.576696+0000","last_change":"2024-09-18T11:54:39.252210+0000","last_active":"2024-09-18T11:54:42.576696+0000","last_peered":"2024-09-18T11:54:42.576696+0000","last_clean":"2024-09-18T11:54:42.576696+0000","last_became_active":"2024-09-18T11:54:18.440195+0000","last_became_peered":"2024-09-18T11:54:18.440195+0000","last_unstale":"2024-09-18T11:54:42.576696+0000","last_undegraded":"2024-09-18T11:54:42.576696+0000","last_fullsized":"2024-09-18T11:54:42.576696+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2111,"log_dups_size":0,"ondisk_log_size":2111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8619218,"num_objects":2106,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2109,"num_write_kb":8420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439949+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:54:18.439949+0000","last_peered":"2024-09-18T11:54:18.439949+0000","last_clean":"2024-09-18T11:54:18.439949+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:54:18.439949+0000","last_undegraded":"2024-09-18T11:54:18.439949+0000","last_fullsized":"2024-09-18T11:54:18.439949+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:14:25.540910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"30'2134","reported_seq":2172,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.522105+0000","last_change":"2024-09-18T11:54:18.445976+0000","last_active":"2024-09-18T11:54:42.522105+0000","last_peered":"2024-09-18T11:54:42.522105+0000","last_clean":"2024-09-18T11:53:53.913371+0000","last_became_active":"2024-09-18T11:54:18.445692+0000","last_became_peered":"2024-09-18T11:54:18.445692+0000","last_unstale":"2024-09-18T11:54:42.522105+0000","last_undegraded":"2024-09-18T11:54:18.433038+0000","last_fullsized":"2024-09-18T11:54:42.522105+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2134,"log_dups_size":0,"ondisk_log_size":2134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8709330,"num_objects":2128,"num_object_clones":0,"num_object_copies":4256,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":8508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2128}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":51,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.441089+0000","last_change":"2024-09-18T11:54:18.441264+0000","last_active":"2024-09-18T11:54:18.441089+0000","last_peered":"2024-09-18T11:54:18.441089+0000","last_clean":"2024-09-18T11:54:03.316651+0000","last_became_active":"2024-09-18T11:54:18.440963+0000","last_became_peered":"2024-09-18T11:54:18.440963+0000","last_unstale":"2024-09-18T11:54:18.441089+0000","last_undegraded":"2024-09-18T11:54:18.432164+0000","last_fullsized":"2024-09-18T11:54:18.441089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:23:22.104360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"30'2183","reported_seq":2211,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.578428+0000","last_change":"2024-09-18T11:54:40.066865+0000","last_active":"2024-09-18T11:54:42.578428+0000","last_peered":"2024-09-18T11:54:42.578428+0000","last_clean":"2024-09-18T11:54:42.578428+0000","last_became_active":"2024-09-18T11:54:18.438876+0000","last_became_peered":"2024-09-18T11:54:18.438876+0000","last_unstale":"2024-09-18T11:54:42.578428+0000","last_undegraded":"2024-09-18T11:54:42.578428+0000","last_fullsized":"2024-09-18T11:54:42.578428+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2183,"log_dups_size":0,"ondisk_log_size":2183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8922322,"num_objects":2180,"num_object_clones":0,"num_object_copies":4360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2181,"num_write_kb":8715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439859+0000","last_change":"2024-09-18T11:54:18.439859+0000","last_active":"2024-09-18T11:54:18.439859+0000","last_peered":"2024-09-18T11:54:18.439859+0000","last_clean":"2024-09-18T11:54:18.439859+0000","last_became_active":"2024-09-18T11:54:18.438620+0000","last_became_peered":"2024-09-18T11:54:18.438620+0000","last_unstale":"2024-09-18T11:54:18.439859+0000","last_undegraded":"2024-09-18T11:54:18.439859+0000","last_fullsized":"2024-09-18T11:54:18.439859+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:09:25.244649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"30'2165","reported_seq":2192,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.574154+0000","last_change":"2024-09-18T11:54:40.474517+0000","last_active":"2024-09-18T11:54:42.574154+0000","last_peered":"2024-09-18T11:54:42.574154+0000","last_clean":"2024-09-18T11:54:42.574154+0000","last_became_active":"2024-09-18T11:54:18.441199+0000","last_became_peered":"2024-09-18T11:54:18.441199+0000","last_unstale":"2024-09-18T11:54:42.574154+0000","last_undegraded":"2024-09-18T11:54:42.574154+0000","last_fullsized":"2024-09-18T11:54:42.574154+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2165,"log_dups_size":0,"ondisk_log_size":2165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8844498,"num_objects":2161,"num_object_clones":0,"num_object_copies":4322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":8639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"30'2146","reported_seq":2188,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.546576+0000","last_change":"2024-09-18T11:54:18.445855+0000","last_active":"2024-09-18T11:54:42.546576+0000","last_peered":"2024-09-18T11:54:42.546576+0000","last_clean":"2024-09-18T11:53:46.243258+0000","last_became_active":"2024-09-18T11:54:18.445615+0000","last_became_peered":"2024-09-18T11:54:18.445615+0000","last_unstale":"2024-09-18T11:54:42.546576+0000","last_undegraded":"2024-09-18T11:54:18.433012+0000","last_fullsized":"2024-09-18T11:54:42.546576+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2146,"log_dups_size":0,"ondisk_log_size":2146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8782441,"num_objects":2145,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":8577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2145}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.442188+0000","last_change":"2024-09-18T11:54:18.442188+0000","last_active":"2024-09-18T11:54:18.442188+0000","last_peered":"2024-09-18T11:54:18.442188+0000","last_clean":"2024-09-18T11:54:18.442188+0000","last_became_active":"2024-09-18T11:54:18.441831+0000","last_became_peered":"2024-09-18T11:54:18.441831+0000","last_unstale":"2024-09-18T11:54:18.442188+0000","last_undegraded":"2024-09-18T11:54:18.442188+0000","last_fullsized":"2024-09-18T11:54:18.442188+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:20:27.929651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"30'2143","reported_seq":2186,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.530783+0000","last_change":"2024-09-18T11:54:18.441937+0000","last_active":"2024-09-18T11:54:42.530783+0000","last_peered":"2024-09-18T11:54:42.530783+0000","last_clean":"2024-09-18T11:53:53.823053+0000","last_became_active":"2024-09-18T11:54:18.441659+0000","last_became_peered":"2024-09-18T11:54:18.441659+0000","last_unstale":"2024-09-18T11:54:42.530783+0000","last_undegraded":"2024-09-18T11:54:18.432961+0000","last_fullsized":"2024-09-18T11:54:42.530783+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2143,"log_dups_size":0,"ondisk_log_size":2143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746811,"num_objects":2138,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":8544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2138}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440580+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:54:18.440580+0000","last_peered":"2024-09-18T11:54:18.440580+0000","last_clean":"2024-09-18T11:54:18.440580+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:54:18.440580+0000","last_undegraded":"2024-09-18T11:54:18.440580+0000","last_fullsized":"2024-09-18T11:54:18.440580+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:10:36.497525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"30'2149","reported_seq":2192,"reported_epoch":30,"state":"active+recovering","last_fresh":"2024-09-18T11:54:42.528497+0000","last_change":"2024-09-18T11:54:26.134693+0000","last_active":"2024-09-18T11:54:42.528497+0000","last_peered":"2024-09-18T11:54:42.528497+0000","last_clean":"2024-09-18T11:53:46.242858+0000","last_became_active":"2024-09-18T11:54:18.440831+0000","last_became_peered":"2024-09-18T11:54:18.440831+0000","last_unstale":"2024-09-18T11:54:42.528497+0000","last_undegraded":"2024-09-18T11:54:42.528497+0000","last_fullsized":"2024-09-18T11:54:42.528497+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2149,"log_dups_size":0,"ondisk_log_size":2149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8787154,"num_objects":2147,"num_object_clones":0,"num_object_copies":4294,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":8582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2147}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441579+0000","last_change":"2024-09-18T11:54:18.441579+0000","last_active":"2024-09-18T11:54:18.441579+0000","last_peered":"2024-09-18T11:54:18.441579+0000","last_clean":"2024-09-18T11:54:18.441579+0000","last_became_active":"2024-09-18T11:54:18.440490+0000","last_became_peered":"2024-09-18T11:54:18.440490+0000","last_unstale":"2024-09-18T11:54:18.441579+0000","last_undegraded":"2024-09-18T11:54:18.441579+0000","last_fullsized":"2024-09-18T11:54:18.441579+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:04.506685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440411+0000","last_change":"2024-09-18T11:54:18.440411+0000","last_active":"2024-09-18T11:54:18.440411+0000","last_peered":"2024-09-18T11:54:18.440411+0000","last_clean":"2024-09-18T11:54:18.440411+0000","last_became_active":"2024-09-18T11:54:18.439759+0000","last_became_peered":"2024-09-18T11:54:18.439759+0000","last_unstale":"2024-09-18T11:54:18.440411+0000","last_undegraded":"2024-09-18T11:54:18.440411+0000","last_fullsized":"2024-09-18T11:54:18.440411+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:02.939140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"30'2100","reported_seq":2143,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.492588+0000","last_change":"2024-09-18T11:54:18.441905+0000","last_active":"2024-09-18T11:54:42.492588+0000","last_peered":"2024-09-18T11:54:42.492588+0000","last_clean":"2024-09-18T11:53:53.582527+0000","last_became_active":"2024-09-18T11:54:18.440844+0000","last_became_peered":"2024-09-18T11:54:18.440844+0000","last_unstale":"2024-09-18T11:54:42.492588+0000","last_undegraded":"2024-09-18T11:54:18.431783+0000","last_fullsized":"2024-09-18T11:54:42.492588+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8554916,"num_objects":2092,"num_object_clones":0,"num_object_copies":4184,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2097,"num_write_kb":8359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2091}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":139215740,"num_objects":34012,"num_object_clones":0,"num_object_copies":68024,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34039,"num_write_kb":135982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278605824,"data_stored":278487538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34066,"ondisk_log_size":34066,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051594,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429056,"kb_used_data":131192,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93942784,"statfs":{"total":96636764160,"available":96197410816,"internally_reserved":0,"allocated":134340608,"data_stored":134164882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.16300000000000001}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247821,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432748,"kb_used_data":134948,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93939092,"statfs":{"total":96636764160,"available":96193630208,"internally_reserved":0,"allocated":138186752,"data_stored":137986075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0190000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139186176,"data_stored":139165302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139304960,"data_stored":139304960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:45.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:45.093 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 19 20 37612 37592 7.72798 5.14453 0.013497 0.0101016 2024-09-18T11:54:45.507 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:45.513 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:54:45.527 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-18T11:54:45.034145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":139215759,"num_objects":34014,"num_object_clones":0,"num_object_copies":68028,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34041,"num_write_kb":135984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34069,"ondisk_log_size":34069,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":861804,"kb_used_data":266140,"kb_used_omap":8,"kb_used_meta":595639,"kb_avail":187881876,"statfs":{"total":193273528320,"available":192391041024,"internally_reserved":0,"allocated":272527360,"data_stored":272150957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9204,"internal_metadata":609934348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,4,9,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":78712832,"num_objects":19217,"num_object_clones":0,"num_object_copies":38434,"num_objects_missing_on_primary":-9,"num_objects_missing":-9,"num_objects_degraded":-18,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":19217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19217,"num_write_kb":76868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003704"},"pg_stats":[{"pgid":"2.f","version":"30'2103","reported_seq":2145,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.545723+0000","last_change":"2024-09-18T11:54:18.444208+0000","last_active":"2024-09-18T11:54:42.545723+0000","last_peered":"2024-09-18T11:54:42.545723+0000","last_clean":"2024-09-18T11:54:42.545723+0000","last_became_active":"2024-09-18T11:54:18.443917+0000","last_became_peered":"2024-09-18T11:54:18.443917+0000","last_unstale":"2024-09-18T11:54:42.545723+0000","last_undegraded":"2024-09-18T11:54:42.545723+0000","last_fullsized":"2024-09-18T11:54:42.545723+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:52:17.496947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8613888,"num_objects":2103,"num_object_clones":0,"num_object_copies":4206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2103,"num_write_kb":8412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'2113","reported_seq":2140,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.563932+0000","last_change":"2024-09-18T11:54:36.889194+0000","last_active":"2024-09-18T11:54:42.563932+0000","last_peered":"2024-09-18T11:54:42.563932+0000","last_clean":"2024-09-18T11:54:42.563932+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:54:42.563932+0000","last_undegraded":"2024-09-18T11:54:42.563932+0000","last_fullsized":"2024-09-18T11:54:42.563932+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8631506,"num_objects":2109,"num_object_clones":0,"num_object_copies":4218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2111,"num_write_kb":8432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"30'2068","reported_seq":2087,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.561204+0000","last_change":"2024-09-18T11:54:37.294288+0000","last_active":"2024-09-18T11:54:42.561204+0000","last_peered":"2024-09-18T11:54:42.561204+0000","last_clean":"2024-09-18T11:54:42.561204+0000","last_became_active":"2024-09-18T11:54:18.439908+0000","last_became_peered":"2024-09-18T11:54:18.439908+0000","last_unstale":"2024-09-18T11:54:42.561204+0000","last_undegraded":"2024-09-18T11:54:42.561204+0000","last_fullsized":"2024-09-18T11:54:42.561204+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2068,"log_dups_size":0,"ondisk_log_size":2068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454761,"num_objects":2065,"num_object_clones":0,"num_object_copies":4130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2067,"num_write_kb":8258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"30'2145","reported_seq":2165,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.566024+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:54:42.566024+0000","last_peered":"2024-09-18T11:54:42.566024+0000","last_clean":"2024-09-18T11:54:42.566024+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:54:42.566024+0000","last_undegraded":"2024-09-18T11:54:42.566024+0000","last_fullsized":"2024-09-18T11:54:42.566024+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2145,"log_dups_size":0,"ondisk_log_size":2145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:35.240042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8785920,"num_objects":2145,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":8580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"30'2088","reported_seq":2137,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.545115+0000","last_change":"2024-09-18T11:54:18.443818+0000","last_active":"2024-09-18T11:54:42.545115+0000","last_peered":"2024-09-18T11:54:42.545115+0000","last_clean":"2024-09-18T11:53:53.013443+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:54:42.545115+0000","last_undegraded":"2024-09-18T11:54:18.431175+0000","last_fullsized":"2024-09-18T11:54:42.545115+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8525010,"num_objects":2083,"num_object_clones":0,"num_object_copies":4166,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":8328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2083}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"30'2129","reported_seq":2173,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.539875+0000","last_change":"2024-09-18T11:54:18.442338+0000","last_active":"2024-09-18T11:54:42.539875+0000","last_peered":"2024-09-18T11:54:42.539875+0000","last_clean":"2024-09-18T11:53:53.869066+0000","last_became_active":"2024-09-18T11:54:18.441285+0000","last_became_peered":"2024-09-18T11:54:18.441285+0000","last_unstale":"2024-09-18T11:54:42.539875+0000","last_undegraded":"2024-09-18T11:54:18.433470+0000","last_fullsized":"2024-09-18T11:54:42.539875+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2129,"log_dups_size":0,"ondisk_log_size":2129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8701138,"num_objects":2126,"num_object_clones":0,"num_object_copies":4252,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2127,"num_write_kb":8499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2126}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"30'2080","reported_seq":2124,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.506365+0000","last_change":"2024-09-18T11:54:18.446054+0000","last_active":"2024-09-18T11:54:42.506365+0000","last_peered":"2024-09-18T11:54:42.506365+0000","last_clean":"2024-09-18T11:53:53.251624+0000","last_became_active":"2024-09-18T11:54:18.445771+0000","last_became_peered":"2024-09-18T11:54:18.445771+0000","last_unstale":"2024-09-18T11:54:42.506365+0000","last_undegraded":"2024-09-18T11:54:18.434522+0000","last_fullsized":"2024-09-18T11:54:42.506365+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2080,"log_dups_size":0,"ondisk_log_size":2080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8496338,"num_objects":2076,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2078,"num_write_kb":8300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2076}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"30'2209","reported_seq":2261,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.522184+0000","last_change":"2024-09-18T11:54:18.446046+0000","last_active":"2024-09-18T11:54:42.522184+0000","last_peered":"2024-09-18T11:54:42.522184+0000","last_clean":"2024-09-18T11:53:43.208142+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:54:42.522184+0000","last_undegraded":"2024-09-18T11:54:18.434814+0000","last_fullsized":"2024-09-18T11:54:42.522184+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2209,"log_dups_size":0,"ondisk_log_size":2209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9040489,"num_objects":2208,"num_object_clones":0,"num_object_copies":4416,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":8829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2208}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441722+0000","last_change":"2024-09-18T11:54:18.441722+0000","last_active":"2024-09-18T11:54:18.441722+0000","last_peered":"2024-09-18T11:54:18.441722+0000","last_clean":"2024-09-18T11:54:18.441722+0000","last_became_active":"2024-09-18T11:54:18.440263+0000","last_became_peered":"2024-09-18T11:54:18.440263+0000","last_unstale":"2024-09-18T11:54:18.441722+0000","last_undegraded":"2024-09-18T11:54:18.441722+0000","last_fullsized":"2024-09-18T11:54:18.441722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:27:26.196691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"30'2111","reported_seq":2131,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.576696+0000","last_change":"2024-09-18T11:54:39.252210+0000","last_active":"2024-09-18T11:54:42.576696+0000","last_peered":"2024-09-18T11:54:42.576696+0000","last_clean":"2024-09-18T11:54:42.576696+0000","last_became_active":"2024-09-18T11:54:18.440195+0000","last_became_peered":"2024-09-18T11:54:18.440195+0000","last_unstale":"2024-09-18T11:54:42.576696+0000","last_undegraded":"2024-09-18T11:54:42.576696+0000","last_fullsized":"2024-09-18T11:54:42.576696+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2111,"log_dups_size":0,"ondisk_log_size":2111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8619218,"num_objects":2106,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2109,"num_write_kb":8420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439949+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:54:18.439949+0000","last_peered":"2024-09-18T11:54:18.439949+0000","last_clean":"2024-09-18T11:54:18.439949+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:54:18.439949+0000","last_undegraded":"2024-09-18T11:54:18.439949+0000","last_fullsized":"2024-09-18T11:54:18.439949+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:14:25.540910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"30'2134","reported_seq":2172,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.522105+0000","last_change":"2024-09-18T11:54:18.445976+0000","last_active":"2024-09-18T11:54:42.522105+0000","last_peered":"2024-09-18T11:54:42.522105+0000","last_clean":"2024-09-18T11:53:53.913371+0000","last_became_active":"2024-09-18T11:54:18.445692+0000","last_became_peered":"2024-09-18T11:54:18.445692+0000","last_unstale":"2024-09-18T11:54:42.522105+0000","last_undegraded":"2024-09-18T11:54:18.433038+0000","last_fullsized":"2024-09-18T11:54:42.522105+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2134,"log_dups_size":0,"ondisk_log_size":2134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8709330,"num_objects":2128,"num_object_clones":0,"num_object_copies":4256,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":8508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2128}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":51,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.441089+0000","last_change":"2024-09-18T11:54:18.441264+0000","last_active":"2024-09-18T11:54:18.441089+0000","last_peered":"2024-09-18T11:54:18.441089+0000","last_clean":"2024-09-18T11:54:03.316651+0000","last_became_active":"2024-09-18T11:54:18.440963+0000","last_became_peered":"2024-09-18T11:54:18.440963+0000","last_unstale":"2024-09-18T11:54:18.441089+0000","last_undegraded":"2024-09-18T11:54:18.432164+0000","last_fullsized":"2024-09-18T11:54:18.441089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:23:22.104360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"30'2183","reported_seq":2211,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.578428+0000","last_change":"2024-09-18T11:54:40.066865+0000","last_active":"2024-09-18T11:54:42.578428+0000","last_peered":"2024-09-18T11:54:42.578428+0000","last_clean":"2024-09-18T11:54:42.578428+0000","last_became_active":"2024-09-18T11:54:18.438876+0000","last_became_peered":"2024-09-18T11:54:18.438876+0000","last_unstale":"2024-09-18T11:54:42.578428+0000","last_undegraded":"2024-09-18T11:54:42.578428+0000","last_fullsized":"2024-09-18T11:54:42.578428+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2183,"log_dups_size":0,"ondisk_log_size":2183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8922322,"num_objects":2180,"num_object_clones":0,"num_object_copies":4360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2181,"num_write_kb":8715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439859+0000","last_change":"2024-09-18T11:54:18.439859+0000","last_active":"2024-09-18T11:54:18.439859+0000","last_peered":"2024-09-18T11:54:18.439859+0000","last_clean":"2024-09-18T11:54:18.439859+0000","last_became_active":"2024-09-18T11:54:18.438620+0000","last_became_peered":"2024-09-18T11:54:18.438620+0000","last_unstale":"2024-09-18T11:54:18.439859+0000","last_undegraded":"2024-09-18T11:54:18.439859+0000","last_fullsized":"2024-09-18T11:54:18.439859+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:09:25.244649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"30'2165","reported_seq":2192,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.574154+0000","last_change":"2024-09-18T11:54:40.474517+0000","last_active":"2024-09-18T11:54:42.574154+0000","last_peered":"2024-09-18T11:54:42.574154+0000","last_clean":"2024-09-18T11:54:42.574154+0000","last_became_active":"2024-09-18T11:54:18.441199+0000","last_became_peered":"2024-09-18T11:54:18.441199+0000","last_unstale":"2024-09-18T11:54:42.574154+0000","last_undegraded":"2024-09-18T11:54:42.574154+0000","last_fullsized":"2024-09-18T11:54:42.574154+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2165,"log_dups_size":0,"ondisk_log_size":2165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8844498,"num_objects":2161,"num_object_clones":0,"num_object_copies":4322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":8639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"30'2146","reported_seq":2188,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.546576+0000","last_change":"2024-09-18T11:54:18.445855+0000","last_active":"2024-09-18T11:54:42.546576+0000","last_peered":"2024-09-18T11:54:42.546576+0000","last_clean":"2024-09-18T11:53:46.243258+0000","last_became_active":"2024-09-18T11:54:18.445615+0000","last_became_peered":"2024-09-18T11:54:18.445615+0000","last_unstale":"2024-09-18T11:54:42.546576+0000","last_undegraded":"2024-09-18T11:54:18.433012+0000","last_fullsized":"2024-09-18T11:54:42.546576+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2146,"log_dups_size":0,"ondisk_log_size":2146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8782441,"num_objects":2145,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":8577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2145}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.442188+0000","last_change":"2024-09-18T11:54:18.442188+0000","last_active":"2024-09-18T11:54:18.442188+0000","last_peered":"2024-09-18T11:54:18.442188+0000","last_clean":"2024-09-18T11:54:18.442188+0000","last_became_active":"2024-09-18T11:54:18.441831+0000","last_became_peered":"2024-09-18T11:54:18.441831+0000","last_unstale":"2024-09-18T11:54:18.442188+0000","last_undegraded":"2024-09-18T11:54:18.442188+0000","last_fullsized":"2024-09-18T11:54:18.442188+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:20:27.929651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"30'2143","reported_seq":2186,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.530783+0000","last_change":"2024-09-18T11:54:18.441937+0000","last_active":"2024-09-18T11:54:42.530783+0000","last_peered":"2024-09-18T11:54:42.530783+0000","last_clean":"2024-09-18T11:53:53.823053+0000","last_became_active":"2024-09-18T11:54:18.441659+0000","last_became_peered":"2024-09-18T11:54:18.441659+0000","last_unstale":"2024-09-18T11:54:42.530783+0000","last_undegraded":"2024-09-18T11:54:18.432961+0000","last_fullsized":"2024-09-18T11:54:42.530783+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2143,"log_dups_size":0,"ondisk_log_size":2143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746811,"num_objects":2138,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":8544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2138}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440580+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:54:18.440580+0000","last_peered":"2024-09-18T11:54:18.440580+0000","last_clean":"2024-09-18T11:54:18.440580+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:54:18.440580+0000","last_undegraded":"2024-09-18T11:54:18.440580+0000","last_fullsized":"2024-09-18T11:54:18.440580+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:10:36.497525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"30'2149","reported_seq":2192,"reported_epoch":30,"state":"active+recovering","last_fresh":"2024-09-18T11:54:42.528497+0000","last_change":"2024-09-18T11:54:26.134693+0000","last_active":"2024-09-18T11:54:42.528497+0000","last_peered":"2024-09-18T11:54:42.528497+0000","last_clean":"2024-09-18T11:53:46.242858+0000","last_became_active":"2024-09-18T11:54:18.440831+0000","last_became_peered":"2024-09-18T11:54:18.440831+0000","last_unstale":"2024-09-18T11:54:42.528497+0000","last_undegraded":"2024-09-18T11:54:42.528497+0000","last_fullsized":"2024-09-18T11:54:42.528497+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2149,"log_dups_size":0,"ondisk_log_size":2149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8787154,"num_objects":2147,"num_object_clones":0,"num_object_copies":4294,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":8582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2147}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441579+0000","last_change":"2024-09-18T11:54:18.441579+0000","last_active":"2024-09-18T11:54:18.441579+0000","last_peered":"2024-09-18T11:54:18.441579+0000","last_clean":"2024-09-18T11:54:18.441579+0000","last_became_active":"2024-09-18T11:54:18.440490+0000","last_became_peered":"2024-09-18T11:54:18.440490+0000","last_unstale":"2024-09-18T11:54:18.441579+0000","last_undegraded":"2024-09-18T11:54:18.441579+0000","last_fullsized":"2024-09-18T11:54:18.441579+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:04.506685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440411+0000","last_change":"2024-09-18T11:54:18.440411+0000","last_active":"2024-09-18T11:54:18.440411+0000","last_peered":"2024-09-18T11:54:18.440411+0000","last_clean":"2024-09-18T11:54:18.440411+0000","last_became_active":"2024-09-18T11:54:18.439759+0000","last_became_peered":"2024-09-18T11:54:18.439759+0000","last_unstale":"2024-09-18T11:54:18.440411+0000","last_undegraded":"2024-09-18T11:54:18.440411+0000","last_fullsized":"2024-09-18T11:54:18.440411+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:02.939140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"30'2100","reported_seq":2143,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.492588+0000","last_change":"2024-09-18T11:54:18.441905+0000","last_active":"2024-09-18T11:54:42.492588+0000","last_peered":"2024-09-18T11:54:42.492588+0000","last_clean":"2024-09-18T11:53:53.582527+0000","last_became_active":"2024-09-18T11:54:18.440844+0000","last_became_peered":"2024-09-18T11:54:18.440844+0000","last_unstale":"2024-09-18T11:54:42.492588+0000","last_undegraded":"2024-09-18T11:54:18.431783+0000","last_fullsized":"2024-09-18T11:54:42.492588+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8554916,"num_objects":2092,"num_object_clones":0,"num_object_copies":4184,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2097,"num_write_kb":8359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2091}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":139215740,"num_objects":34012,"num_object_clones":0,"num_object_copies":68024,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34039,"num_write_kb":135982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278605824,"data_stored":278487538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34066,"ondisk_log_size":34066,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051594,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429056,"kb_used_data":131192,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93942784,"statfs":{"total":96636764160,"available":96197410816,"internally_reserved":0,"allocated":134340608,"data_stored":134164882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.16300000000000001}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247821,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432748,"kb_used_data":134948,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93939092,"statfs":{"total":96636764160,"available":96193630208,"internally_reserved":0,"allocated":138186752,"data_stored":137986075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0190000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139186176,"data_stored":139165302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139304960,"data_stored":139304960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:45.529 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:45.944 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:45.956 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:54:45.965 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-18T11:54:45.034145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":139215759,"num_objects":34014,"num_object_clones":0,"num_object_copies":68028,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34041,"num_write_kb":135984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34069,"ondisk_log_size":34069,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":861804,"kb_used_data":266140,"kb_used_omap":8,"kb_used_meta":595639,"kb_avail":187881876,"statfs":{"total":193273528320,"available":192391041024,"internally_reserved":0,"allocated":272527360,"data_stored":272150957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9204,"internal_metadata":609934348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,4,9,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":78712832,"num_objects":19217,"num_object_clones":0,"num_object_copies":38434,"num_objects_missing_on_primary":-9,"num_objects_missing":-9,"num_objects_degraded":-18,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":19217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19217,"num_write_kb":76868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003704"},"pg_stats":[{"pgid":"2.f","version":"30'2103","reported_seq":2145,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.545723+0000","last_change":"2024-09-18T11:54:18.444208+0000","last_active":"2024-09-18T11:54:42.545723+0000","last_peered":"2024-09-18T11:54:42.545723+0000","last_clean":"2024-09-18T11:54:42.545723+0000","last_became_active":"2024-09-18T11:54:18.443917+0000","last_became_peered":"2024-09-18T11:54:18.443917+0000","last_unstale":"2024-09-18T11:54:42.545723+0000","last_undegraded":"2024-09-18T11:54:42.545723+0000","last_fullsized":"2024-09-18T11:54:42.545723+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:52:17.496947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8613888,"num_objects":2103,"num_object_clones":0,"num_object_copies":4206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2103,"num_write_kb":8412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'2113","reported_seq":2140,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.563932+0000","last_change":"2024-09-18T11:54:36.889194+0000","last_active":"2024-09-18T11:54:42.563932+0000","last_peered":"2024-09-18T11:54:42.563932+0000","last_clean":"2024-09-18T11:54:42.563932+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:54:42.563932+0000","last_undegraded":"2024-09-18T11:54:42.563932+0000","last_fullsized":"2024-09-18T11:54:42.563932+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8631506,"num_objects":2109,"num_object_clones":0,"num_object_copies":4218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2111,"num_write_kb":8432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"30'2068","reported_seq":2087,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.561204+0000","last_change":"2024-09-18T11:54:37.294288+0000","last_active":"2024-09-18T11:54:42.561204+0000","last_peered":"2024-09-18T11:54:42.561204+0000","last_clean":"2024-09-18T11:54:42.561204+0000","last_became_active":"2024-09-18T11:54:18.439908+0000","last_became_peered":"2024-09-18T11:54:18.439908+0000","last_unstale":"2024-09-18T11:54:42.561204+0000","last_undegraded":"2024-09-18T11:54:42.561204+0000","last_fullsized":"2024-09-18T11:54:42.561204+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2068,"log_dups_size":0,"ondisk_log_size":2068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454761,"num_objects":2065,"num_object_clones":0,"num_object_copies":4130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2067,"num_write_kb":8258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"30'2145","reported_seq":2165,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.566024+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:54:42.566024+0000","last_peered":"2024-09-18T11:54:42.566024+0000","last_clean":"2024-09-18T11:54:42.566024+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:54:42.566024+0000","last_undegraded":"2024-09-18T11:54:42.566024+0000","last_fullsized":"2024-09-18T11:54:42.566024+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2145,"log_dups_size":0,"ondisk_log_size":2145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:35.240042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8785920,"num_objects":2145,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":8580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"30'2088","reported_seq":2137,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.545115+0000","last_change":"2024-09-18T11:54:18.443818+0000","last_active":"2024-09-18T11:54:42.545115+0000","last_peered":"2024-09-18T11:54:42.545115+0000","last_clean":"2024-09-18T11:53:53.013443+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:54:42.545115+0000","last_undegraded":"2024-09-18T11:54:18.431175+0000","last_fullsized":"2024-09-18T11:54:42.545115+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8525010,"num_objects":2083,"num_object_clones":0,"num_object_copies":4166,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":8328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2083}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"30'2129","reported_seq":2173,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.539875+0000","last_change":"2024-09-18T11:54:18.442338+0000","last_active":"2024-09-18T11:54:42.539875+0000","last_peered":"2024-09-18T11:54:42.539875+0000","last_clean":"2024-09-18T11:53:53.869066+0000","last_became_active":"2024-09-18T11:54:18.441285+0000","last_became_peered":"2024-09-18T11:54:18.441285+0000","last_unstale":"2024-09-18T11:54:42.539875+0000","last_undegraded":"2024-09-18T11:54:18.433470+0000","last_fullsized":"2024-09-18T11:54:42.539875+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2129,"log_dups_size":0,"ondisk_log_size":2129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8701138,"num_objects":2126,"num_object_clones":0,"num_object_copies":4252,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2127,"num_write_kb":8499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2126}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"30'2080","reported_seq":2124,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.506365+0000","last_change":"2024-09-18T11:54:18.446054+0000","last_active":"2024-09-18T11:54:42.506365+0000","last_peered":"2024-09-18T11:54:42.506365+0000","last_clean":"2024-09-18T11:53:53.251624+0000","last_became_active":"2024-09-18T11:54:18.445771+0000","last_became_peered":"2024-09-18T11:54:18.445771+0000","last_unstale":"2024-09-18T11:54:42.506365+0000","last_undegraded":"2024-09-18T11:54:18.434522+0000","last_fullsized":"2024-09-18T11:54:42.506365+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2080,"log_dups_size":0,"ondisk_log_size":2080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8496338,"num_objects":2076,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2078,"num_write_kb":8300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2076}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"30'2209","reported_seq":2261,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.522184+0000","last_change":"2024-09-18T11:54:18.446046+0000","last_active":"2024-09-18T11:54:42.522184+0000","last_peered":"2024-09-18T11:54:42.522184+0000","last_clean":"2024-09-18T11:53:43.208142+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:54:42.522184+0000","last_undegraded":"2024-09-18T11:54:18.434814+0000","last_fullsized":"2024-09-18T11:54:42.522184+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2209,"log_dups_size":0,"ondisk_log_size":2209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9040489,"num_objects":2208,"num_object_clones":0,"num_object_copies":4416,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":8829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2208}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441722+0000","last_change":"2024-09-18T11:54:18.441722+0000","last_active":"2024-09-18T11:54:18.441722+0000","last_peered":"2024-09-18T11:54:18.441722+0000","last_clean":"2024-09-18T11:54:18.441722+0000","last_became_active":"2024-09-18T11:54:18.440263+0000","last_became_peered":"2024-09-18T11:54:18.440263+0000","last_unstale":"2024-09-18T11:54:18.441722+0000","last_undegraded":"2024-09-18T11:54:18.441722+0000","last_fullsized":"2024-09-18T11:54:18.441722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:27:26.196691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"30'2111","reported_seq":2131,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.576696+0000","last_change":"2024-09-18T11:54:39.252210+0000","last_active":"2024-09-18T11:54:42.576696+0000","last_peered":"2024-09-18T11:54:42.576696+0000","last_clean":"2024-09-18T11:54:42.576696+0000","last_became_active":"2024-09-18T11:54:18.440195+0000","last_became_peered":"2024-09-18T11:54:18.440195+0000","last_unstale":"2024-09-18T11:54:42.576696+0000","last_undegraded":"2024-09-18T11:54:42.576696+0000","last_fullsized":"2024-09-18T11:54:42.576696+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2111,"log_dups_size":0,"ondisk_log_size":2111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8619218,"num_objects":2106,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2109,"num_write_kb":8420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439949+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:54:18.439949+0000","last_peered":"2024-09-18T11:54:18.439949+0000","last_clean":"2024-09-18T11:54:18.439949+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:54:18.439949+0000","last_undegraded":"2024-09-18T11:54:18.439949+0000","last_fullsized":"2024-09-18T11:54:18.439949+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:14:25.540910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"30'2134","reported_seq":2172,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.522105+0000","last_change":"2024-09-18T11:54:18.445976+0000","last_active":"2024-09-18T11:54:42.522105+0000","last_peered":"2024-09-18T11:54:42.522105+0000","last_clean":"2024-09-18T11:53:53.913371+0000","last_became_active":"2024-09-18T11:54:18.445692+0000","last_became_peered":"2024-09-18T11:54:18.445692+0000","last_unstale":"2024-09-18T11:54:42.522105+0000","last_undegraded":"2024-09-18T11:54:18.433038+0000","last_fullsized":"2024-09-18T11:54:42.522105+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2134,"log_dups_size":0,"ondisk_log_size":2134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8709330,"num_objects":2128,"num_object_clones":0,"num_object_copies":4256,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":8508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2128}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":51,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.441089+0000","last_change":"2024-09-18T11:54:18.441264+0000","last_active":"2024-09-18T11:54:18.441089+0000","last_peered":"2024-09-18T11:54:18.441089+0000","last_clean":"2024-09-18T11:54:03.316651+0000","last_became_active":"2024-09-18T11:54:18.440963+0000","last_became_peered":"2024-09-18T11:54:18.440963+0000","last_unstale":"2024-09-18T11:54:18.441089+0000","last_undegraded":"2024-09-18T11:54:18.432164+0000","last_fullsized":"2024-09-18T11:54:18.441089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:23:22.104360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"30'2183","reported_seq":2211,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.578428+0000","last_change":"2024-09-18T11:54:40.066865+0000","last_active":"2024-09-18T11:54:42.578428+0000","last_peered":"2024-09-18T11:54:42.578428+0000","last_clean":"2024-09-18T11:54:42.578428+0000","last_became_active":"2024-09-18T11:54:18.438876+0000","last_became_peered":"2024-09-18T11:54:18.438876+0000","last_unstale":"2024-09-18T11:54:42.578428+0000","last_undegraded":"2024-09-18T11:54:42.578428+0000","last_fullsized":"2024-09-18T11:54:42.578428+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2183,"log_dups_size":0,"ondisk_log_size":2183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8922322,"num_objects":2180,"num_object_clones":0,"num_object_copies":4360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2181,"num_write_kb":8715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439859+0000","last_change":"2024-09-18T11:54:18.439859+0000","last_active":"2024-09-18T11:54:18.439859+0000","last_peered":"2024-09-18T11:54:18.439859+0000","last_clean":"2024-09-18T11:54:18.439859+0000","last_became_active":"2024-09-18T11:54:18.438620+0000","last_became_peered":"2024-09-18T11:54:18.438620+0000","last_unstale":"2024-09-18T11:54:18.439859+0000","last_undegraded":"2024-09-18T11:54:18.439859+0000","last_fullsized":"2024-09-18T11:54:18.439859+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:09:25.244649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"30'2165","reported_seq":2192,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:42.574154+0000","last_change":"2024-09-18T11:54:40.474517+0000","last_active":"2024-09-18T11:54:42.574154+0000","last_peered":"2024-09-18T11:54:42.574154+0000","last_clean":"2024-09-18T11:54:42.574154+0000","last_became_active":"2024-09-18T11:54:18.441199+0000","last_became_peered":"2024-09-18T11:54:18.441199+0000","last_unstale":"2024-09-18T11:54:42.574154+0000","last_undegraded":"2024-09-18T11:54:42.574154+0000","last_fullsized":"2024-09-18T11:54:42.574154+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2165,"log_dups_size":0,"ondisk_log_size":2165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8844498,"num_objects":2161,"num_object_clones":0,"num_object_copies":4322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":8639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"30'2146","reported_seq":2188,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.546576+0000","last_change":"2024-09-18T11:54:18.445855+0000","last_active":"2024-09-18T11:54:42.546576+0000","last_peered":"2024-09-18T11:54:42.546576+0000","last_clean":"2024-09-18T11:53:46.243258+0000","last_became_active":"2024-09-18T11:54:18.445615+0000","last_became_peered":"2024-09-18T11:54:18.445615+0000","last_unstale":"2024-09-18T11:54:42.546576+0000","last_undegraded":"2024-09-18T11:54:18.433012+0000","last_fullsized":"2024-09-18T11:54:42.546576+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2146,"log_dups_size":0,"ondisk_log_size":2146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8782441,"num_objects":2145,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":8577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2145}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.442188+0000","last_change":"2024-09-18T11:54:18.442188+0000","last_active":"2024-09-18T11:54:18.442188+0000","last_peered":"2024-09-18T11:54:18.442188+0000","last_clean":"2024-09-18T11:54:18.442188+0000","last_became_active":"2024-09-18T11:54:18.441831+0000","last_became_peered":"2024-09-18T11:54:18.441831+0000","last_unstale":"2024-09-18T11:54:18.442188+0000","last_undegraded":"2024-09-18T11:54:18.442188+0000","last_fullsized":"2024-09-18T11:54:18.442188+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:20:27.929651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"30'2143","reported_seq":2186,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.530783+0000","last_change":"2024-09-18T11:54:18.441937+0000","last_active":"2024-09-18T11:54:42.530783+0000","last_peered":"2024-09-18T11:54:42.530783+0000","last_clean":"2024-09-18T11:53:53.823053+0000","last_became_active":"2024-09-18T11:54:18.441659+0000","last_became_peered":"2024-09-18T11:54:18.441659+0000","last_unstale":"2024-09-18T11:54:42.530783+0000","last_undegraded":"2024-09-18T11:54:18.432961+0000","last_fullsized":"2024-09-18T11:54:42.530783+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2143,"log_dups_size":0,"ondisk_log_size":2143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746811,"num_objects":2138,"num_object_clones":0,"num_object_copies":4276,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":8544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2138}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440580+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:54:18.440580+0000","last_peered":"2024-09-18T11:54:18.440580+0000","last_clean":"2024-09-18T11:54:18.440580+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:54:18.440580+0000","last_undegraded":"2024-09-18T11:54:18.440580+0000","last_fullsized":"2024-09-18T11:54:18.440580+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:10:36.497525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"30'2149","reported_seq":2192,"reported_epoch":30,"state":"active+recovering","last_fresh":"2024-09-18T11:54:42.528497+0000","last_change":"2024-09-18T11:54:26.134693+0000","last_active":"2024-09-18T11:54:42.528497+0000","last_peered":"2024-09-18T11:54:42.528497+0000","last_clean":"2024-09-18T11:53:46.242858+0000","last_became_active":"2024-09-18T11:54:18.440831+0000","last_became_peered":"2024-09-18T11:54:18.440831+0000","last_unstale":"2024-09-18T11:54:42.528497+0000","last_undegraded":"2024-09-18T11:54:42.528497+0000","last_fullsized":"2024-09-18T11:54:42.528497+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2149,"log_dups_size":0,"ondisk_log_size":2149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8787154,"num_objects":2147,"num_object_clones":0,"num_object_copies":4294,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":8582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2147}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441579+0000","last_change":"2024-09-18T11:54:18.441579+0000","last_active":"2024-09-18T11:54:18.441579+0000","last_peered":"2024-09-18T11:54:18.441579+0000","last_clean":"2024-09-18T11:54:18.441579+0000","last_became_active":"2024-09-18T11:54:18.440490+0000","last_became_peered":"2024-09-18T11:54:18.440490+0000","last_unstale":"2024-09-18T11:54:18.441579+0000","last_undegraded":"2024-09-18T11:54:18.441579+0000","last_fullsized":"2024-09-18T11:54:18.441579+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:04.506685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440411+0000","last_change":"2024-09-18T11:54:18.440411+0000","last_active":"2024-09-18T11:54:18.440411+0000","last_peered":"2024-09-18T11:54:18.440411+0000","last_clean":"2024-09-18T11:54:18.440411+0000","last_became_active":"2024-09-18T11:54:18.439759+0000","last_became_peered":"2024-09-18T11:54:18.439759+0000","last_unstale":"2024-09-18T11:54:18.440411+0000","last_undegraded":"2024-09-18T11:54:18.440411+0000","last_fullsized":"2024-09-18T11:54:18.440411+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:02.939140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"30'2100","reported_seq":2143,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:42.492588+0000","last_change":"2024-09-18T11:54:18.441905+0000","last_active":"2024-09-18T11:54:42.492588+0000","last_peered":"2024-09-18T11:54:42.492588+0000","last_clean":"2024-09-18T11:53:53.582527+0000","last_became_active":"2024-09-18T11:54:18.440844+0000","last_became_peered":"2024-09-18T11:54:18.440844+0000","last_unstale":"2024-09-18T11:54:42.492588+0000","last_undegraded":"2024-09-18T11:54:18.431783+0000","last_fullsized":"2024-09-18T11:54:42.492588+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8554916,"num_objects":2092,"num_object_clones":0,"num_object_copies":4184,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2097,"num_write_kb":8359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2091}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":139215740,"num_objects":34012,"num_object_clones":0,"num_object_copies":68024,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34039,"num_write_kb":135982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278605824,"data_stored":278487538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34066,"ondisk_log_size":34066,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051594,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429056,"kb_used_data":131192,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93942784,"statfs":{"total":96636764160,"available":96197410816,"internally_reserved":0,"allocated":134340608,"data_stored":134164882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.16300000000000001}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247821,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":432748,"kb_used_data":134948,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93939092,"statfs":{"total":96636764160,"available":96193630208,"internally_reserved":0,"allocated":138186752,"data_stored":137986075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0190000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139186176,"data_stored":139165302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139304960,"data_stored":139304960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:46.094 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:2024-09-18T11:54:46.093787+0000 min lat: 0.0021114 max lat: 0.0661932 avg lat: 0.0102677 2024-09-18T11:54:46.094 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:54:46.094 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 20 20 38956 38936 7.60407 5.25 0.0167468 0.0102677 2024-09-18T11:54:47.094 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 21 20 40477 40457 7.52487 5.94141 0.0106085 0.0103769 2024-09-18T11:54:48.094 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 22 20 41970 41950 7.4479 5.83203 0.0162372 0.0104831 2024-09-18T11:54:48.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:49.095 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 23 20 43427 43407 7.37101 5.69141 0.0188066 0.0105914 2024-09-18T11:54:49.352 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:49.383 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:54:49.384 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-09-18T11:54:49.034902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":163492751,"num_objects":39941,"num_object_clones":0,"num_object_copies":79882,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39968,"num_write_kb":159692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39996,"ondisk_log_size":39996,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":928764,"kb_used_data":316460,"kb_used_omap":8,"kb_used_meta":612279,"kb_avail":187814916,"statfs":{"total":193273528320,"available":192322473984,"internally_reserved":0,"allocated":324055040,"data_stored":323678637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9204,"internal_metadata":626973708},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,7,10,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":76210176,"num_objects":18606,"num_object_clones":0,"num_object_copies":37212,"num_objects_missing_on_primary":-9,"num_objects_missing":-9,"num_objects_degraded":-18,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":18606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18606,"num_write_kb":74424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003393"},"pg_stats":[{"pgid":"2.f","version":"30'2466","reported_seq":2508,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:46.319967+0000","last_change":"2024-09-18T11:54:18.444208+0000","last_active":"2024-09-18T11:54:46.319967+0000","last_peered":"2024-09-18T11:54:46.319967+0000","last_clean":"2024-09-18T11:54:46.319967+0000","last_became_active":"2024-09-18T11:54:18.443917+0000","last_became_peered":"2024-09-18T11:54:18.443917+0000","last_unstale":"2024-09-18T11:54:46.319967+0000","last_undegraded":"2024-09-18T11:54:46.319967+0000","last_fullsized":"2024-09-18T11:54:46.319967+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2466,"log_dups_size":0,"ondisk_log_size":2466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:52:17.496947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10100736,"num_objects":2466,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2466,"num_write_kb":9864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'2584","reported_seq":2611,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.396618+0000","last_change":"2024-09-18T11:54:36.889194+0000","last_active":"2024-09-18T11:54:47.396618+0000","last_peered":"2024-09-18T11:54:47.396618+0000","last_clean":"2024-09-18T11:54:47.396618+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:54:47.396618+0000","last_undegraded":"2024-09-18T11:54:47.396618+0000","last_fullsized":"2024-09-18T11:54:47.396618+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2584,"log_dups_size":0,"ondisk_log_size":2584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10560722,"num_objects":2580,"num_object_clones":0,"num_object_copies":5160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2582,"num_write_kb":10316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"30'2505","reported_seq":2525,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.413574+0000","last_change":"2024-09-18T11:54:37.294288+0000","last_active":"2024-09-18T11:54:47.413574+0000","last_peered":"2024-09-18T11:54:47.413574+0000","last_clean":"2024-09-18T11:54:47.413574+0000","last_became_active":"2024-09-18T11:54:18.439908+0000","last_became_peered":"2024-09-18T11:54:18.439908+0000","last_unstale":"2024-09-18T11:54:47.413574+0000","last_undegraded":"2024-09-18T11:54:47.413574+0000","last_fullsized":"2024-09-18T11:54:47.413574+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2505,"log_dups_size":0,"ondisk_log_size":2505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10244713,"num_objects":2502,"num_object_clones":0,"num_object_copies":5004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":10006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"30'2559","reported_seq":2579,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.418029+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:54:47.418029+0000","last_peered":"2024-09-18T11:54:47.418029+0000","last_clean":"2024-09-18T11:54:47.418029+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:54:47.418029+0000","last_undegraded":"2024-09-18T11:54:47.418029+0000","last_fullsized":"2024-09-18T11:54:47.418029+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2559,"log_dups_size":0,"ondisk_log_size":2559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:35.240042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10481664,"num_objects":2559,"num_object_clones":0,"num_object_copies":5118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":10236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"30'2402","reported_seq":2451,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.303233+0000","last_change":"2024-09-18T11:54:18.443818+0000","last_active":"2024-09-18T11:54:46.303233+0000","last_peered":"2024-09-18T11:54:46.303233+0000","last_clean":"2024-09-18T11:53:53.013443+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:54:46.303233+0000","last_undegraded":"2024-09-18T11:54:18.431175+0000","last_fullsized":"2024-09-18T11:54:46.303233+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2402,"log_dups_size":0,"ondisk_log_size":2402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9811154,"num_objects":2397,"num_object_clones":0,"num_object_copies":4794,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2400,"num_write_kb":9584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2397}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"30'2451","reported_seq":2494,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.317817+0000","last_change":"2024-09-18T11:54:18.442338+0000","last_active":"2024-09-18T11:54:46.317817+0000","last_peered":"2024-09-18T11:54:46.317817+0000","last_clean":"2024-09-18T11:53:53.869066+0000","last_became_active":"2024-09-18T11:54:18.441285+0000","last_became_peered":"2024-09-18T11:54:18.441285+0000","last_unstale":"2024-09-18T11:54:46.317817+0000","last_undegraded":"2024-09-18T11:54:18.433470+0000","last_fullsized":"2024-09-18T11:54:46.317817+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2451,"log_dups_size":0,"ondisk_log_size":2451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10020050,"num_objects":2448,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":9787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2448}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"30'2412","reported_seq":2456,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.289483+0000","last_change":"2024-09-18T11:54:18.446054+0000","last_active":"2024-09-18T11:54:46.289483+0000","last_peered":"2024-09-18T11:54:46.289483+0000","last_clean":"2024-09-18T11:53:53.251624+0000","last_became_active":"2024-09-18T11:54:18.445771+0000","last_became_peered":"2024-09-18T11:54:18.445771+0000","last_unstale":"2024-09-18T11:54:46.289483+0000","last_undegraded":"2024-09-18T11:54:18.434522+0000","last_fullsized":"2024-09-18T11:54:46.289483+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2412,"log_dups_size":0,"ondisk_log_size":2412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9856210,"num_objects":2408,"num_object_clones":0,"num_object_copies":4816,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2410,"num_write_kb":9628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2408}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"30'2549","reported_seq":2601,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.304214+0000","last_change":"2024-09-18T11:54:18.446046+0000","last_active":"2024-09-18T11:54:46.304214+0000","last_peered":"2024-09-18T11:54:46.304214+0000","last_clean":"2024-09-18T11:53:43.208142+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:54:46.304214+0000","last_undegraded":"2024-09-18T11:54:18.434814+0000","last_fullsized":"2024-09-18T11:54:46.304214+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2549,"log_dups_size":0,"ondisk_log_size":2549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10433129,"num_objects":2548,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2548,"num_write_kb":10189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2548}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441722+0000","last_change":"2024-09-18T11:54:18.441722+0000","last_active":"2024-09-18T11:54:18.441722+0000","last_peered":"2024-09-18T11:54:18.441722+0000","last_clean":"2024-09-18T11:54:18.441722+0000","last_became_active":"2024-09-18T11:54:18.440263+0000","last_became_peered":"2024-09-18T11:54:18.440263+0000","last_unstale":"2024-09-18T11:54:18.441722+0000","last_undegraded":"2024-09-18T11:54:18.441722+0000","last_fullsized":"2024-09-18T11:54:18.441722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:27:26.196691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"30'2542","reported_seq":2562,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.422853+0000","last_change":"2024-09-18T11:54:39.252210+0000","last_active":"2024-09-18T11:54:47.422853+0000","last_peered":"2024-09-18T11:54:47.422853+0000","last_clean":"2024-09-18T11:54:47.422853+0000","last_became_active":"2024-09-18T11:54:18.440195+0000","last_became_peered":"2024-09-18T11:54:18.440195+0000","last_unstale":"2024-09-18T11:54:47.422853+0000","last_undegraded":"2024-09-18T11:54:47.422853+0000","last_fullsized":"2024-09-18T11:54:47.422853+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2542,"log_dups_size":0,"ondisk_log_size":2542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10384594,"num_objects":2537,"num_object_clones":0,"num_object_copies":5074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2540,"num_write_kb":10144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439949+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:54:18.439949+0000","last_peered":"2024-09-18T11:54:18.439949+0000","last_clean":"2024-09-18T11:54:18.439949+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:54:18.439949+0000","last_undegraded":"2024-09-18T11:54:18.439949+0000","last_fullsized":"2024-09-18T11:54:18.439949+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:14:25.540910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"30'2471","reported_seq":2509,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.302551+0000","last_change":"2024-09-18T11:54:18.445976+0000","last_active":"2024-09-18T11:54:46.302551+0000","last_peered":"2024-09-18T11:54:46.302551+0000","last_clean":"2024-09-18T11:53:53.913371+0000","last_became_active":"2024-09-18T11:54:18.445692+0000","last_became_peered":"2024-09-18T11:54:18.445692+0000","last_unstale":"2024-09-18T11:54:46.302551+0000","last_undegraded":"2024-09-18T11:54:18.433038+0000","last_fullsized":"2024-09-18T11:54:46.302551+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2471,"log_dups_size":0,"ondisk_log_size":2471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10089682,"num_objects":2465,"num_object_clones":0,"num_object_copies":4930,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2469,"num_write_kb":9856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2465}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":51,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.441089+0000","last_change":"2024-09-18T11:54:18.441264+0000","last_active":"2024-09-18T11:54:18.441089+0000","last_peered":"2024-09-18T11:54:18.441089+0000","last_clean":"2024-09-18T11:54:03.316651+0000","last_became_active":"2024-09-18T11:54:18.440963+0000","last_became_peered":"2024-09-18T11:54:18.440963+0000","last_unstale":"2024-09-18T11:54:18.441089+0000","last_undegraded":"2024-09-18T11:54:18.432164+0000","last_fullsized":"2024-09-18T11:54:18.441089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:23:22.104360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"30'2582","reported_seq":2610,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.355931+0000","last_change":"2024-09-18T11:54:40.066865+0000","last_active":"2024-09-18T11:54:47.355931+0000","last_peered":"2024-09-18T11:54:47.355931+0000","last_clean":"2024-09-18T11:54:47.355931+0000","last_became_active":"2024-09-18T11:54:18.438876+0000","last_became_peered":"2024-09-18T11:54:18.438876+0000","last_unstale":"2024-09-18T11:54:47.355931+0000","last_undegraded":"2024-09-18T11:54:47.355931+0000","last_fullsized":"2024-09-18T11:54:47.355931+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2582,"log_dups_size":0,"ondisk_log_size":2582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10556626,"num_objects":2579,"num_object_clones":0,"num_object_copies":5158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2580,"num_write_kb":10311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439859+0000","last_change":"2024-09-18T11:54:18.439859+0000","last_active":"2024-09-18T11:54:18.439859+0000","last_peered":"2024-09-18T11:54:18.439859+0000","last_clean":"2024-09-18T11:54:18.439859+0000","last_became_active":"2024-09-18T11:54:18.438620+0000","last_became_peered":"2024-09-18T11:54:18.438620+0000","last_unstale":"2024-09-18T11:54:18.439859+0000","last_undegraded":"2024-09-18T11:54:18.439859+0000","last_fullsized":"2024-09-18T11:54:18.439859+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:09:25.244649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"30'2616","reported_seq":2641,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.430574+0000","last_change":"2024-09-18T11:54:40.474517+0000","last_active":"2024-09-18T11:54:47.430574+0000","last_peered":"2024-09-18T11:54:47.430574+0000","last_clean":"2024-09-18T11:54:47.430574+0000","last_became_active":"2024-09-18T11:54:18.441199+0000","last_became_peered":"2024-09-18T11:54:18.441199+0000","last_unstale":"2024-09-18T11:54:47.430574+0000","last_undegraded":"2024-09-18T11:54:47.430574+0000","last_fullsized":"2024-09-18T11:54:47.430574+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2616,"log_dups_size":0,"ondisk_log_size":2616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10691794,"num_objects":2612,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2614,"num_write_kb":10443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"30'2456","reported_seq":2498,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.304031+0000","last_change":"2024-09-18T11:54:18.445855+0000","last_active":"2024-09-18T11:54:46.304031+0000","last_peered":"2024-09-18T11:54:46.304031+0000","last_clean":"2024-09-18T11:53:46.243258+0000","last_became_active":"2024-09-18T11:54:18.445615+0000","last_became_peered":"2024-09-18T11:54:18.445615+0000","last_unstale":"2024-09-18T11:54:46.304031+0000","last_undegraded":"2024-09-18T11:54:18.433012+0000","last_fullsized":"2024-09-18T11:54:46.304031+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2456,"log_dups_size":0,"ondisk_log_size":2456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10052201,"num_objects":2455,"num_object_clones":0,"num_object_copies":4910,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2455,"num_write_kb":9817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2455}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.442188+0000","last_change":"2024-09-18T11:54:18.442188+0000","last_active":"2024-09-18T11:54:18.442188+0000","last_peered":"2024-09-18T11:54:18.442188+0000","last_clean":"2024-09-18T11:54:18.442188+0000","last_became_active":"2024-09-18T11:54:18.441831+0000","last_became_peered":"2024-09-18T11:54:18.441831+0000","last_unstale":"2024-09-18T11:54:18.442188+0000","last_undegraded":"2024-09-18T11:54:18.442188+0000","last_fullsized":"2024-09-18T11:54:18.442188+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:20:27.929651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"30'2455","reported_seq":2498,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.308696+0000","last_change":"2024-09-18T11:54:18.441937+0000","last_active":"2024-09-18T11:54:46.308696+0000","last_peered":"2024-09-18T11:54:46.308696+0000","last_clean":"2024-09-18T11:53:53.823053+0000","last_became_active":"2024-09-18T11:54:18.441659+0000","last_became_peered":"2024-09-18T11:54:18.441659+0000","last_unstale":"2024-09-18T11:54:46.308696+0000","last_undegraded":"2024-09-18T11:54:18.432961+0000","last_fullsized":"2024-09-18T11:54:46.308696+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2455,"log_dups_size":0,"ondisk_log_size":2455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10024763,"num_objects":2450,"num_object_clones":0,"num_object_copies":4900,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2450}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440580+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:54:18.440580+0000","last_peered":"2024-09-18T11:54:18.440580+0000","last_clean":"2024-09-18T11:54:18.440580+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:54:18.440580+0000","last_undegraded":"2024-09-18T11:54:18.440580+0000","last_fullsized":"2024-09-18T11:54:18.440580+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:10:36.497525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"30'2506","reported_seq":2550,"reported_epoch":30,"state":"active+recovering","last_fresh":"2024-09-18T11:54:46.301324+0000","last_change":"2024-09-18T11:54:26.134693+0000","last_active":"2024-09-18T11:54:46.301324+0000","last_peered":"2024-09-18T11:54:46.301324+0000","last_clean":"2024-09-18T11:53:46.242858+0000","last_became_active":"2024-09-18T11:54:18.440831+0000","last_became_peered":"2024-09-18T11:54:18.440831+0000","last_unstale":"2024-09-18T11:54:46.301324+0000","last_undegraded":"2024-09-18T11:54:46.301324+0000","last_fullsized":"2024-09-18T11:54:46.301324+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2506,"log_dups_size":0,"ondisk_log_size":2506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10249426,"num_objects":2504,"num_object_clones":0,"num_object_copies":5008,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":10010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2504}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441579+0000","last_change":"2024-09-18T11:54:18.441579+0000","last_active":"2024-09-18T11:54:18.441579+0000","last_peered":"2024-09-18T11:54:18.441579+0000","last_clean":"2024-09-18T11:54:18.441579+0000","last_became_active":"2024-09-18T11:54:18.440490+0000","last_became_peered":"2024-09-18T11:54:18.440490+0000","last_unstale":"2024-09-18T11:54:18.441579+0000","last_undegraded":"2024-09-18T11:54:18.441579+0000","last_fullsized":"2024-09-18T11:54:18.441579+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:04.506685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440411+0000","last_change":"2024-09-18T11:54:18.440411+0000","last_active":"2024-09-18T11:54:18.440411+0000","last_peered":"2024-09-18T11:54:18.440411+0000","last_clean":"2024-09-18T11:54:18.440411+0000","last_became_active":"2024-09-18T11:54:18.439759+0000","last_became_peered":"2024-09-18T11:54:18.439759+0000","last_unstale":"2024-09-18T11:54:18.440411+0000","last_undegraded":"2024-09-18T11:54:18.440411+0000","last_fullsized":"2024-09-18T11:54:18.440411+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:02.939140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"30'2437","reported_seq":2480,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.320431+0000","last_change":"2024-09-18T11:54:18.441905+0000","last_active":"2024-09-18T11:54:46.320431+0000","last_peered":"2024-09-18T11:54:46.320431+0000","last_clean":"2024-09-18T11:53:53.582527+0000","last_became_active":"2024-09-18T11:54:18.440844+0000","last_became_peered":"2024-09-18T11:54:18.440844+0000","last_unstale":"2024-09-18T11:54:46.320431+0000","last_undegraded":"2024-09-18T11:54:18.431783+0000","last_fullsized":"2024-09-18T11:54:46.320431+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2437,"log_dups_size":0,"ondisk_log_size":2437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9935268,"num_objects":2429,"num_object_clones":0,"num_object_copies":4858,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2434,"num_write_kb":9707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2428}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":163492732,"num_objects":39939,"num_object_clones":0,"num_object_copies":79878,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39966,"num_write_kb":159690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329035776,"data_stored":328917490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39993,"ondisk_log_size":39993,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051595,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465280,"kb_used_data":159096,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93906560,"statfs":{"total":96636764160,"available":96160317440,"internally_reserved":0,"allocated":162914304,"data_stored":162738578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,3,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247822,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463484,"kb_used_data":157364,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93908356,"statfs":{"total":96636764160,"available":96162156544,"internally_reserved":0,"allocated":161140736,"data_stored":160940059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161042432,"data_stored":161021558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167878656,"data_stored":167878656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:49.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:49.818 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:49.831 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:54:49.838 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-09-18T11:54:49.034902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":163492751,"num_objects":39941,"num_object_clones":0,"num_object_copies":79882,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39968,"num_write_kb":159692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39996,"ondisk_log_size":39996,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":928764,"kb_used_data":316460,"kb_used_omap":8,"kb_used_meta":612279,"kb_avail":187814916,"statfs":{"total":193273528320,"available":192322473984,"internally_reserved":0,"allocated":324055040,"data_stored":323678637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9204,"internal_metadata":626973708},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,7,10,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":76210176,"num_objects":18606,"num_object_clones":0,"num_object_copies":37212,"num_objects_missing_on_primary":-9,"num_objects_missing":-9,"num_objects_degraded":-18,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":18606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18606,"num_write_kb":74424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003393"},"pg_stats":[{"pgid":"2.f","version":"30'2466","reported_seq":2508,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:46.319967+0000","last_change":"2024-09-18T11:54:18.444208+0000","last_active":"2024-09-18T11:54:46.319967+0000","last_peered":"2024-09-18T11:54:46.319967+0000","last_clean":"2024-09-18T11:54:46.319967+0000","last_became_active":"2024-09-18T11:54:18.443917+0000","last_became_peered":"2024-09-18T11:54:18.443917+0000","last_unstale":"2024-09-18T11:54:46.319967+0000","last_undegraded":"2024-09-18T11:54:46.319967+0000","last_fullsized":"2024-09-18T11:54:46.319967+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2466,"log_dups_size":0,"ondisk_log_size":2466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:52:17.496947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10100736,"num_objects":2466,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2466,"num_write_kb":9864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'2584","reported_seq":2611,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.396618+0000","last_change":"2024-09-18T11:54:36.889194+0000","last_active":"2024-09-18T11:54:47.396618+0000","last_peered":"2024-09-18T11:54:47.396618+0000","last_clean":"2024-09-18T11:54:47.396618+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:54:47.396618+0000","last_undegraded":"2024-09-18T11:54:47.396618+0000","last_fullsized":"2024-09-18T11:54:47.396618+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2584,"log_dups_size":0,"ondisk_log_size":2584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10560722,"num_objects":2580,"num_object_clones":0,"num_object_copies":5160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2582,"num_write_kb":10316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"30'2505","reported_seq":2525,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.413574+0000","last_change":"2024-09-18T11:54:37.294288+0000","last_active":"2024-09-18T11:54:47.413574+0000","last_peered":"2024-09-18T11:54:47.413574+0000","last_clean":"2024-09-18T11:54:47.413574+0000","last_became_active":"2024-09-18T11:54:18.439908+0000","last_became_peered":"2024-09-18T11:54:18.439908+0000","last_unstale":"2024-09-18T11:54:47.413574+0000","last_undegraded":"2024-09-18T11:54:47.413574+0000","last_fullsized":"2024-09-18T11:54:47.413574+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2505,"log_dups_size":0,"ondisk_log_size":2505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10244713,"num_objects":2502,"num_object_clones":0,"num_object_copies":5004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":10006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"30'2559","reported_seq":2579,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.418029+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:54:47.418029+0000","last_peered":"2024-09-18T11:54:47.418029+0000","last_clean":"2024-09-18T11:54:47.418029+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:54:47.418029+0000","last_undegraded":"2024-09-18T11:54:47.418029+0000","last_fullsized":"2024-09-18T11:54:47.418029+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2559,"log_dups_size":0,"ondisk_log_size":2559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:35.240042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10481664,"num_objects":2559,"num_object_clones":0,"num_object_copies":5118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":10236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"30'2402","reported_seq":2451,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.303233+0000","last_change":"2024-09-18T11:54:18.443818+0000","last_active":"2024-09-18T11:54:46.303233+0000","last_peered":"2024-09-18T11:54:46.303233+0000","last_clean":"2024-09-18T11:53:53.013443+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:54:46.303233+0000","last_undegraded":"2024-09-18T11:54:18.431175+0000","last_fullsized":"2024-09-18T11:54:46.303233+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2402,"log_dups_size":0,"ondisk_log_size":2402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9811154,"num_objects":2397,"num_object_clones":0,"num_object_copies":4794,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2400,"num_write_kb":9584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2397}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"30'2451","reported_seq":2494,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.317817+0000","last_change":"2024-09-18T11:54:18.442338+0000","last_active":"2024-09-18T11:54:46.317817+0000","last_peered":"2024-09-18T11:54:46.317817+0000","last_clean":"2024-09-18T11:53:53.869066+0000","last_became_active":"2024-09-18T11:54:18.441285+0000","last_became_peered":"2024-09-18T11:54:18.441285+0000","last_unstale":"2024-09-18T11:54:46.317817+0000","last_undegraded":"2024-09-18T11:54:18.433470+0000","last_fullsized":"2024-09-18T11:54:46.317817+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2451,"log_dups_size":0,"ondisk_log_size":2451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10020050,"num_objects":2448,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":9787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2448}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"30'2412","reported_seq":2456,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.289483+0000","last_change":"2024-09-18T11:54:18.446054+0000","last_active":"2024-09-18T11:54:46.289483+0000","last_peered":"2024-09-18T11:54:46.289483+0000","last_clean":"2024-09-18T11:53:53.251624+0000","last_became_active":"2024-09-18T11:54:18.445771+0000","last_became_peered":"2024-09-18T11:54:18.445771+0000","last_unstale":"2024-09-18T11:54:46.289483+0000","last_undegraded":"2024-09-18T11:54:18.434522+0000","last_fullsized":"2024-09-18T11:54:46.289483+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2412,"log_dups_size":0,"ondisk_log_size":2412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9856210,"num_objects":2408,"num_object_clones":0,"num_object_copies":4816,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2410,"num_write_kb":9628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2408}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"30'2549","reported_seq":2601,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.304214+0000","last_change":"2024-09-18T11:54:18.446046+0000","last_active":"2024-09-18T11:54:46.304214+0000","last_peered":"2024-09-18T11:54:46.304214+0000","last_clean":"2024-09-18T11:53:43.208142+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:54:46.304214+0000","last_undegraded":"2024-09-18T11:54:18.434814+0000","last_fullsized":"2024-09-18T11:54:46.304214+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2549,"log_dups_size":0,"ondisk_log_size":2549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10433129,"num_objects":2548,"num_object_clones":0,"num_object_copies":5096,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2548,"num_write_kb":10189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2548}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441722+0000","last_change":"2024-09-18T11:54:18.441722+0000","last_active":"2024-09-18T11:54:18.441722+0000","last_peered":"2024-09-18T11:54:18.441722+0000","last_clean":"2024-09-18T11:54:18.441722+0000","last_became_active":"2024-09-18T11:54:18.440263+0000","last_became_peered":"2024-09-18T11:54:18.440263+0000","last_unstale":"2024-09-18T11:54:18.441722+0000","last_undegraded":"2024-09-18T11:54:18.441722+0000","last_fullsized":"2024-09-18T11:54:18.441722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:27:26.196691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"30'2542","reported_seq":2562,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.422853+0000","last_change":"2024-09-18T11:54:39.252210+0000","last_active":"2024-09-18T11:54:47.422853+0000","last_peered":"2024-09-18T11:54:47.422853+0000","last_clean":"2024-09-18T11:54:47.422853+0000","last_became_active":"2024-09-18T11:54:18.440195+0000","last_became_peered":"2024-09-18T11:54:18.440195+0000","last_unstale":"2024-09-18T11:54:47.422853+0000","last_undegraded":"2024-09-18T11:54:47.422853+0000","last_fullsized":"2024-09-18T11:54:47.422853+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2542,"log_dups_size":0,"ondisk_log_size":2542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10384594,"num_objects":2537,"num_object_clones":0,"num_object_copies":5074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2540,"num_write_kb":10144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439949+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:54:18.439949+0000","last_peered":"2024-09-18T11:54:18.439949+0000","last_clean":"2024-09-18T11:54:18.439949+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:54:18.439949+0000","last_undegraded":"2024-09-18T11:54:18.439949+0000","last_fullsized":"2024-09-18T11:54:18.439949+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:14:25.540910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"30'2471","reported_seq":2509,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.302551+0000","last_change":"2024-09-18T11:54:18.445976+0000","last_active":"2024-09-18T11:54:46.302551+0000","last_peered":"2024-09-18T11:54:46.302551+0000","last_clean":"2024-09-18T11:53:53.913371+0000","last_became_active":"2024-09-18T11:54:18.445692+0000","last_became_peered":"2024-09-18T11:54:18.445692+0000","last_unstale":"2024-09-18T11:54:46.302551+0000","last_undegraded":"2024-09-18T11:54:18.433038+0000","last_fullsized":"2024-09-18T11:54:46.302551+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2471,"log_dups_size":0,"ondisk_log_size":2471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10089682,"num_objects":2465,"num_object_clones":0,"num_object_copies":4930,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2469,"num_write_kb":9856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2465}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":51,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:18.441089+0000","last_change":"2024-09-18T11:54:18.441264+0000","last_active":"2024-09-18T11:54:18.441089+0000","last_peered":"2024-09-18T11:54:18.441089+0000","last_clean":"2024-09-18T11:54:03.316651+0000","last_became_active":"2024-09-18T11:54:18.440963+0000","last_became_peered":"2024-09-18T11:54:18.440963+0000","last_unstale":"2024-09-18T11:54:18.441089+0000","last_undegraded":"2024-09-18T11:54:18.432164+0000","last_fullsized":"2024-09-18T11:54:18.441089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:23:22.104360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"30'2582","reported_seq":2610,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.355931+0000","last_change":"2024-09-18T11:54:40.066865+0000","last_active":"2024-09-18T11:54:47.355931+0000","last_peered":"2024-09-18T11:54:47.355931+0000","last_clean":"2024-09-18T11:54:47.355931+0000","last_became_active":"2024-09-18T11:54:18.438876+0000","last_became_peered":"2024-09-18T11:54:18.438876+0000","last_unstale":"2024-09-18T11:54:47.355931+0000","last_undegraded":"2024-09-18T11:54:47.355931+0000","last_fullsized":"2024-09-18T11:54:47.355931+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2582,"log_dups_size":0,"ondisk_log_size":2582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10556626,"num_objects":2579,"num_object_clones":0,"num_object_copies":5158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2580,"num_write_kb":10311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439859+0000","last_change":"2024-09-18T11:54:18.439859+0000","last_active":"2024-09-18T11:54:18.439859+0000","last_peered":"2024-09-18T11:54:18.439859+0000","last_clean":"2024-09-18T11:54:18.439859+0000","last_became_active":"2024-09-18T11:54:18.438620+0000","last_became_peered":"2024-09-18T11:54:18.438620+0000","last_unstale":"2024-09-18T11:54:18.439859+0000","last_undegraded":"2024-09-18T11:54:18.439859+0000","last_fullsized":"2024-09-18T11:54:18.439859+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:09:25.244649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"30'2616","reported_seq":2641,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.430574+0000","last_change":"2024-09-18T11:54:40.474517+0000","last_active":"2024-09-18T11:54:47.430574+0000","last_peered":"2024-09-18T11:54:47.430574+0000","last_clean":"2024-09-18T11:54:47.430574+0000","last_became_active":"2024-09-18T11:54:18.441199+0000","last_became_peered":"2024-09-18T11:54:18.441199+0000","last_unstale":"2024-09-18T11:54:47.430574+0000","last_undegraded":"2024-09-18T11:54:47.430574+0000","last_fullsized":"2024-09-18T11:54:47.430574+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2616,"log_dups_size":0,"ondisk_log_size":2616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10691794,"num_objects":2612,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2614,"num_write_kb":10443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"30'2456","reported_seq":2498,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.304031+0000","last_change":"2024-09-18T11:54:18.445855+0000","last_active":"2024-09-18T11:54:46.304031+0000","last_peered":"2024-09-18T11:54:46.304031+0000","last_clean":"2024-09-18T11:53:46.243258+0000","last_became_active":"2024-09-18T11:54:18.445615+0000","last_became_peered":"2024-09-18T11:54:18.445615+0000","last_unstale":"2024-09-18T11:54:46.304031+0000","last_undegraded":"2024-09-18T11:54:18.433012+0000","last_fullsized":"2024-09-18T11:54:46.304031+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2456,"log_dups_size":0,"ondisk_log_size":2456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10052201,"num_objects":2455,"num_object_clones":0,"num_object_copies":4910,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2455,"num_write_kb":9817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2455}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.442188+0000","last_change":"2024-09-18T11:54:18.442188+0000","last_active":"2024-09-18T11:54:18.442188+0000","last_peered":"2024-09-18T11:54:18.442188+0000","last_clean":"2024-09-18T11:54:18.442188+0000","last_became_active":"2024-09-18T11:54:18.441831+0000","last_became_peered":"2024-09-18T11:54:18.441831+0000","last_unstale":"2024-09-18T11:54:18.442188+0000","last_undegraded":"2024-09-18T11:54:18.442188+0000","last_fullsized":"2024-09-18T11:54:18.442188+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:20:27.929651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"30'2455","reported_seq":2498,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.308696+0000","last_change":"2024-09-18T11:54:18.441937+0000","last_active":"2024-09-18T11:54:46.308696+0000","last_peered":"2024-09-18T11:54:46.308696+0000","last_clean":"2024-09-18T11:53:53.823053+0000","last_became_active":"2024-09-18T11:54:18.441659+0000","last_became_peered":"2024-09-18T11:54:18.441659+0000","last_unstale":"2024-09-18T11:54:46.308696+0000","last_undegraded":"2024-09-18T11:54:18.432961+0000","last_fullsized":"2024-09-18T11:54:46.308696+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2455,"log_dups_size":0,"ondisk_log_size":2455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10024763,"num_objects":2450,"num_object_clones":0,"num_object_copies":4900,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2450}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440580+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:54:18.440580+0000","last_peered":"2024-09-18T11:54:18.440580+0000","last_clean":"2024-09-18T11:54:18.440580+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:54:18.440580+0000","last_undegraded":"2024-09-18T11:54:18.440580+0000","last_fullsized":"2024-09-18T11:54:18.440580+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:10:36.497525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"30'2506","reported_seq":2550,"reported_epoch":30,"state":"active+recovering","last_fresh":"2024-09-18T11:54:46.301324+0000","last_change":"2024-09-18T11:54:26.134693+0000","last_active":"2024-09-18T11:54:46.301324+0000","last_peered":"2024-09-18T11:54:46.301324+0000","last_clean":"2024-09-18T11:53:46.242858+0000","last_became_active":"2024-09-18T11:54:18.440831+0000","last_became_peered":"2024-09-18T11:54:18.440831+0000","last_unstale":"2024-09-18T11:54:46.301324+0000","last_undegraded":"2024-09-18T11:54:46.301324+0000","last_fullsized":"2024-09-18T11:54:46.301324+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2506,"log_dups_size":0,"ondisk_log_size":2506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10249426,"num_objects":2504,"num_object_clones":0,"num_object_copies":5008,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":10010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2504}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441579+0000","last_change":"2024-09-18T11:54:18.441579+0000","last_active":"2024-09-18T11:54:18.441579+0000","last_peered":"2024-09-18T11:54:18.441579+0000","last_clean":"2024-09-18T11:54:18.441579+0000","last_became_active":"2024-09-18T11:54:18.440490+0000","last_became_peered":"2024-09-18T11:54:18.440490+0000","last_unstale":"2024-09-18T11:54:18.441579+0000","last_undegraded":"2024-09-18T11:54:18.441579+0000","last_fullsized":"2024-09-18T11:54:18.441579+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:04.506685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440411+0000","last_change":"2024-09-18T11:54:18.440411+0000","last_active":"2024-09-18T11:54:18.440411+0000","last_peered":"2024-09-18T11:54:18.440411+0000","last_clean":"2024-09-18T11:54:18.440411+0000","last_became_active":"2024-09-18T11:54:18.439759+0000","last_became_peered":"2024-09-18T11:54:18.439759+0000","last_unstale":"2024-09-18T11:54:18.440411+0000","last_undegraded":"2024-09-18T11:54:18.440411+0000","last_fullsized":"2024-09-18T11:54:18.440411+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:02.939140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"30'2437","reported_seq":2480,"reported_epoch":30,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-18T11:54:46.320431+0000","last_change":"2024-09-18T11:54:18.441905+0000","last_active":"2024-09-18T11:54:46.320431+0000","last_peered":"2024-09-18T11:54:46.320431+0000","last_clean":"2024-09-18T11:53:53.582527+0000","last_became_active":"2024-09-18T11:54:18.440844+0000","last_became_peered":"2024-09-18T11:54:18.440844+0000","last_unstale":"2024-09-18T11:54:46.320431+0000","last_undegraded":"2024-09-18T11:54:18.431783+0000","last_fullsized":"2024-09-18T11:54:46.320431+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2437,"log_dups_size":0,"ondisk_log_size":2437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9935268,"num_objects":2429,"num_object_clones":0,"num_object_copies":4858,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2434,"num_write_kb":9707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2428}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":163492732,"num_objects":39939,"num_object_clones":0,"num_object_copies":79878,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39966,"num_write_kb":159690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329035776,"data_stored":328917490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39993,"ondisk_log_size":39993,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051595,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465280,"kb_used_data":159096,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93906560,"statfs":{"total":96636764160,"available":96160317440,"internally_reserved":0,"allocated":162914304,"data_stored":162738578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,3,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247822,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463484,"kb_used_data":157364,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93908356,"statfs":{"total":96636764160,"available":96162156544,"internally_reserved":0,"allocated":161140736,"data_stored":160940059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161042432,"data_stored":161021558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167878656,"data_stored":167878656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:50.095 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 24 20 44709 44689 7.27253 5.00781 0.00928404 0.0107355 2024-09-18T11:54:51.095 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 25 20 46029 46009 7.18786 5.15625 0.00776267 0.0108612 2024-09-18T11:54:52.097 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 26 20 47399 47379 7.11687 5.35156 0.00951775 0.0109704 2024-09-18T11:54:52.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:54:53.097 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 27 20 48733 48713 7.04627 5.21094 0.0143963 0.0110797 2024-09-18T11:54:53.244 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:54:53.249 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:54:53.263 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-09-18T11:54:53.035664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":192130749,"num_objects":46931,"num_object_clones":0,"num_object_copies":93862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44154,"num_write_kb":176448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46988,"ondisk_log_size":46988,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":963592,"kb_used_data":367928,"kb_used_omap":8,"kb_used_meta":595639,"kb_avail":187780088,"statfs":{"total":193273528320,"available":192286810112,"internally_reserved":0,"allocated":376758272,"data_stored":376391708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9204,"internal_metadata":609934348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,6,5,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":89873198,"num_objects":21940,"num_object_clones":0,"num_object_copies":43880,"num_objects_missing_on_primary":-24,"num_objects_missing":-24,"num_objects_degraded":-46,"num_objects_misplaced":0,"num_objects_unfound":-24,"num_objects_dirty":21940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19136,"num_write_kb":76556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003031"},"pg_stats":[{"pgid":"2.f","version":"30'2919","reported_seq":2961,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.319810+0000","last_change":"2024-09-18T11:54:18.444208+0000","last_active":"2024-09-18T11:54:51.319810+0000","last_peered":"2024-09-18T11:54:51.319810+0000","last_clean":"2024-09-18T11:54:51.319810+0000","last_became_active":"2024-09-18T11:54:18.443917+0000","last_became_peered":"2024-09-18T11:54:18.443917+0000","last_unstale":"2024-09-18T11:54:51.319810+0000","last_undegraded":"2024-09-18T11:54:51.319810+0000","last_fullsized":"2024-09-18T11:54:51.319810+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2919,"log_dups_size":0,"ondisk_log_size":2919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:52:17.496947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11956224,"num_objects":2919,"num_object_clones":0,"num_object_copies":5838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2919,"num_write_kb":11676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"30'2981","reported_seq":3130,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:52.426347+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:54:52.426347+0000","last_peered":"2024-09-18T11:54:52.426347+0000","last_clean":"2024-09-18T11:54:52.426347+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:54:52.426347+0000","last_undegraded":"2024-09-18T11:54:52.426347+0000","last_fullsized":"2024-09-18T11:54:52.426347+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:15:49.048464+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12185600,"num_objects":2975,"num_object_clones":0,"num_object_copies":5950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":173,"num_write_kb":692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"30'2958","reported_seq":2977,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:52.435659+0000","last_change":"2024-09-18T11:54:37.294288+0000","last_active":"2024-09-18T11:54:52.435659+0000","last_peered":"2024-09-18T11:54:52.435659+0000","last_clean":"2024-09-18T11:54:52.435659+0000","last_became_active":"2024-09-18T11:54:18.439908+0000","last_became_peered":"2024-09-18T11:54:18.439908+0000","last_unstale":"2024-09-18T11:54:52.435659+0000","last_undegraded":"2024-09-18T11:54:52.435659+0000","last_fullsized":"2024-09-18T11:54:52.435659+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2958,"log_dups_size":0,"ondisk_log_size":2958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12100201,"num_objects":2955,"num_object_clones":0,"num_object_copies":5910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2957,"num_write_kb":11818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"30'2989","reported_seq":3009,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:52.426100+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:54:52.426100+0000","last_peered":"2024-09-18T11:54:52.426100+0000","last_clean":"2024-09-18T11:54:52.426100+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:54:52.426100+0000","last_undegraded":"2024-09-18T11:54:52.426100+0000","last_fullsized":"2024-09-18T11:54:52.426100+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2989,"log_dups_size":0,"ondisk_log_size":2989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:25:35.240042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12242944,"num_objects":2989,"num_object_clones":0,"num_object_copies":5978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2989,"num_write_kb":11956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"30'2858","reported_seq":2996,"reported_epoch":30,"state":"active+clean+scrubbing","last_fresh":"2024-09-18T11:54:51.321811+0000","last_change":"2024-09-18T11:54:50.381469+0000","last_active":"2024-09-18T11:54:51.321811+0000","last_peered":"2024-09-18T11:54:51.321811+0000","last_clean":"2024-09-18T11:54:51.321811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:54:51.321811+0000","last_undegraded":"2024-09-18T11:54:51.321811+0000","last_fullsized":"2024-09-18T11:54:51.321811+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":1874,"log_size":2858,"log_dups_size":0,"ondisk_log_size":2858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11678930,"num_objects":2853,"num_object_clones":0,"num_object_copies":5706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"30'2894","reported_seq":2948,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.304877+0000","last_change":"2024-09-18T11:54:47.619070+0000","last_active":"2024-09-18T11:54:51.304877+0000","last_peered":"2024-09-18T11:54:51.304877+0000","last_clean":"2024-09-18T11:54:51.304877+0000","last_became_active":"2024-09-18T11:54:18.441285+0000","last_became_peered":"2024-09-18T11:54:18.441285+0000","last_unstale":"2024-09-18T11:54:51.304877+0000","last_undegraded":"2024-09-18T11:54:51.304877+0000","last_fullsized":"2024-09-18T11:54:51.304877+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2894,"log_dups_size":0,"ondisk_log_size":2894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11834578,"num_objects":2891,"num_object_clones":0,"num_object_copies":5782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2892,"num_write_kb":11559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"30'2887","reported_seq":2940,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.316872+0000","last_change":"2024-09-18T11:54:47.891782+0000","last_active":"2024-09-18T11:54:51.316872+0000","last_peered":"2024-09-18T11:54:51.316872+0000","last_clean":"2024-09-18T11:54:51.316872+0000","last_became_active":"2024-09-18T11:54:18.445771+0000","last_became_peered":"2024-09-18T11:54:18.445771+0000","last_unstale":"2024-09-18T11:54:51.316872+0000","last_undegraded":"2024-09-18T11:54:51.316872+0000","last_fullsized":"2024-09-18T11:54:51.316872+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2887,"log_dups_size":0,"ondisk_log_size":2887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11801810,"num_objects":2883,"num_object_clones":0,"num_object_copies":5766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2885,"num_write_kb":11528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"30'2998","reported_seq":3057,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.290222+0000","last_change":"2024-09-18T11:54:47.832713+0000","last_active":"2024-09-18T11:54:51.290222+0000","last_peered":"2024-09-18T11:54:51.290222+0000","last_clean":"2024-09-18T11:54:51.290222+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:54:51.290222+0000","last_undegraded":"2024-09-18T11:54:51.290222+0000","last_fullsized":"2024-09-18T11:54:51.290222+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2998,"log_dups_size":0,"ondisk_log_size":2998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12272233,"num_objects":2997,"num_object_clones":0,"num_object_copies":5994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2997,"num_write_kb":11985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441722+0000","last_change":"2024-09-18T11:54:18.441722+0000","last_active":"2024-09-18T11:54:18.441722+0000","last_peered":"2024-09-18T11:54:18.441722+0000","last_clean":"2024-09-18T11:54:18.441722+0000","last_became_active":"2024-09-18T11:54:18.440263+0000","last_became_peered":"2024-09-18T11:54:18.440263+0000","last_unstale":"2024-09-18T11:54:18.441722+0000","last_undegraded":"2024-09-18T11:54:18.441722+0000","last_fullsized":"2024-09-18T11:54:18.441722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:27:26.196691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"30'2948","reported_seq":2968,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:52.435804+0000","last_change":"2024-09-18T11:54:39.252210+0000","last_active":"2024-09-18T11:54:52.435804+0000","last_peered":"2024-09-18T11:54:52.435804+0000","last_clean":"2024-09-18T11:54:52.435804+0000","last_became_active":"2024-09-18T11:54:18.440195+0000","last_became_peered":"2024-09-18T11:54:18.440195+0000","last_unstale":"2024-09-18T11:54:52.435804+0000","last_undegraded":"2024-09-18T11:54:52.435804+0000","last_fullsized":"2024-09-18T11:54:52.435804+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2948,"log_dups_size":0,"ondisk_log_size":2948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12047570,"num_objects":2943,"num_object_clones":0,"num_object_copies":5886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2946,"num_write_kb":11768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439949+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:54:18.439949+0000","last_peered":"2024-09-18T11:54:18.439949+0000","last_clean":"2024-09-18T11:54:18.439949+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:54:18.439949+0000","last_undegraded":"2024-09-18T11:54:18.439949+0000","last_fullsized":"2024-09-18T11:54:18.439949+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:14:25.540910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"30'2878","reported_seq":2925,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.312823+0000","last_change":"2024-09-18T11:54:47.799684+0000","last_active":"2024-09-18T11:54:51.312823+0000","last_peered":"2024-09-18T11:54:51.312823+0000","last_clean":"2024-09-18T11:54:51.312823+0000","last_became_active":"2024-09-18T11:54:18.445692+0000","last_became_peered":"2024-09-18T11:54:18.445692+0000","last_unstale":"2024-09-18T11:54:51.312823+0000","last_undegraded":"2024-09-18T11:54:51.312823+0000","last_fullsized":"2024-09-18T11:54:51.312823+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2878,"log_dups_size":0,"ondisk_log_size":2878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11756754,"num_objects":2872,"num_object_clones":0,"num_object_copies":5744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2876,"num_write_kb":11484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:47.905229+0000","last_change":"2024-09-18T11:54:47.905229+0000","last_active":"2024-09-18T11:54:47.905229+0000","last_peered":"2024-09-18T11:54:47.905229+0000","last_clean":"2024-09-18T11:54:47.905229+0000","last_became_active":"2024-09-18T11:54:18.440963+0000","last_became_peered":"2024-09-18T11:54:18.440963+0000","last_unstale":"2024-09-18T11:54:47.905229+0000","last_undegraded":"2024-09-18T11:54:47.905229+0000","last_fullsized":"2024-09-18T11:54:47.905229+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:23:22.104360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"30'3022","reported_seq":3050,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:52.410432+0000","last_change":"2024-09-18T11:54:40.066865+0000","last_active":"2024-09-18T11:54:52.410432+0000","last_peered":"2024-09-18T11:54:52.410432+0000","last_clean":"2024-09-18T11:54:52.410432+0000","last_became_active":"2024-09-18T11:54:18.438876+0000","last_became_peered":"2024-09-18T11:54:18.438876+0000","last_unstale":"2024-09-18T11:54:52.410432+0000","last_undegraded":"2024-09-18T11:54:52.410432+0000","last_fullsized":"2024-09-18T11:54:52.410432+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3022,"log_dups_size":0,"ondisk_log_size":3022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12358866,"num_objects":3019,"num_object_clones":0,"num_object_copies":6038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":12071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.439859+0000","last_change":"2024-09-18T11:54:18.439859+0000","last_active":"2024-09-18T11:54:18.439859+0000","last_peered":"2024-09-18T11:54:18.439859+0000","last_clean":"2024-09-18T11:54:18.439859+0000","last_became_active":"2024-09-18T11:54:18.438620+0000","last_became_peered":"2024-09-18T11:54:18.438620+0000","last_unstale":"2024-09-18T11:54:18.439859+0000","last_undegraded":"2024-09-18T11:54:18.439859+0000","last_fullsized":"2024-09-18T11:54:18.439859+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:09:25.244649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"30'3063","reported_seq":3090,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:52.410719+0000","last_change":"2024-09-18T11:54:40.474517+0000","last_active":"2024-09-18T11:54:52.410719+0000","last_peered":"2024-09-18T11:54:52.410719+0000","last_clean":"2024-09-18T11:54:52.410719+0000","last_became_active":"2024-09-18T11:54:18.441199+0000","last_became_peered":"2024-09-18T11:54:18.441199+0000","last_unstale":"2024-09-18T11:54:52.410719+0000","last_undegraded":"2024-09-18T11:54:52.410719+0000","last_fullsized":"2024-09-18T11:54:52.410719+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3063,"log_dups_size":0,"ondisk_log_size":3063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12522706,"num_objects":3059,"num_object_clones":0,"num_object_copies":6118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3061,"num_write_kb":12231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"30'2891","reported_seq":2942,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.321450+0000","last_change":"2024-09-18T11:54:47.710861+0000","last_active":"2024-09-18T11:54:51.321450+0000","last_peered":"2024-09-18T11:54:51.321450+0000","last_clean":"2024-09-18T11:54:51.321450+0000","last_became_active":"2024-09-18T11:54:18.445615+0000","last_became_peered":"2024-09-18T11:54:18.445615+0000","last_unstale":"2024-09-18T11:54:51.321450+0000","last_undegraded":"2024-09-18T11:54:51.321450+0000","last_fullsized":"2024-09-18T11:54:51.321450+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2891,"log_dups_size":0,"ondisk_log_size":2891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11833961,"num_objects":2890,"num_object_clones":0,"num_object_copies":5780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2890,"num_write_kb":11557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.442188+0000","last_change":"2024-09-18T11:54:18.442188+0000","last_active":"2024-09-18T11:54:18.442188+0000","last_peered":"2024-09-18T11:54:18.442188+0000","last_clean":"2024-09-18T11:54:18.442188+0000","last_became_active":"2024-09-18T11:54:18.441831+0000","last_became_peered":"2024-09-18T11:54:18.441831+0000","last_unstale":"2024-09-18T11:54:18.442188+0000","last_undegraded":"2024-09-18T11:54:18.442188+0000","last_fullsized":"2024-09-18T11:54:18.442188+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:20:27.929651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"30'2909","reported_seq":2964,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.319438+0000","last_change":"2024-09-18T11:54:47.549248+0000","last_active":"2024-09-18T11:54:51.319438+0000","last_peered":"2024-09-18T11:54:51.319438+0000","last_clean":"2024-09-18T11:54:51.319438+0000","last_became_active":"2024-09-18T11:54:18.441659+0000","last_became_peered":"2024-09-18T11:54:18.441659+0000","last_unstale":"2024-09-18T11:54:51.319438+0000","last_undegraded":"2024-09-18T11:54:51.319438+0000","last_fullsized":"2024-09-18T11:54:51.319438+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2909,"log_dups_size":0,"ondisk_log_size":2909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11884347,"num_objects":2904,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2906,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440580+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:54:18.440580+0000","last_peered":"2024-09-18T11:54:18.440580+0000","last_clean":"2024-09-18T11:54:18.440580+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:54:18.440580+0000","last_undegraded":"2024-09-18T11:54:18.440580+0000","last_fullsized":"2024-09-18T11:54:18.440580+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:10:36.497525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"30'2937","reported_seq":2990,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.312271+0000","last_change":"2024-09-18T11:54:47.381093+0000","last_active":"2024-09-18T11:54:51.312271+0000","last_peered":"2024-09-18T11:54:51.312271+0000","last_clean":"2024-09-18T11:54:51.312271+0000","last_became_active":"2024-09-18T11:54:18.440831+0000","last_became_peered":"2024-09-18T11:54:18.440831+0000","last_unstale":"2024-09-18T11:54:51.312271+0000","last_undegraded":"2024-09-18T11:54:51.312271+0000","last_fullsized":"2024-09-18T11:54:51.312271+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2937,"log_dups_size":0,"ondisk_log_size":2937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12014802,"num_objects":2935,"num_object_clones":0,"num_object_copies":5870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":11734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.441579+0000","last_change":"2024-09-18T11:54:18.441579+0000","last_active":"2024-09-18T11:54:18.441579+0000","last_peered":"2024-09-18T11:54:18.441579+0000","last_clean":"2024-09-18T11:54:18.441579+0000","last_became_active":"2024-09-18T11:54:18.440490+0000","last_became_peered":"2024-09-18T11:54:18.440490+0000","last_unstale":"2024-09-18T11:54:18.441579+0000","last_undegraded":"2024-09-18T11:54:18.441579+0000","last_fullsized":"2024-09-18T11:54:18.441579+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:38:04.506685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:18.440411+0000","last_change":"2024-09-18T11:54:18.440411+0000","last_active":"2024-09-18T11:54:18.440411+0000","last_peered":"2024-09-18T11:54:18.440411+0000","last_clean":"2024-09-18T11:54:18.440411+0000","last_became_active":"2024-09-18T11:54:18.439759+0000","last_became_peered":"2024-09-18T11:54:18.439759+0000","last_unstale":"2024-09-18T11:54:18.440411+0000","last_undegraded":"2024-09-18T11:54:18.440411+0000","last_fullsized":"2024-09-18T11:54:18.440411+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:22:02.939140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"30'2853","reported_seq":2911,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-18T11:54:51.303644+0000","last_change":"2024-09-18T11:54:47.459484+0000","last_active":"2024-09-18T11:54:51.303644+0000","last_peered":"2024-09-18T11:54:51.303644+0000","last_clean":"2024-09-18T11:54:51.303644+0000","last_became_active":"2024-09-18T11:54:18.440844+0000","last_became_peered":"2024-09-18T11:54:18.440844+0000","last_unstale":"2024-09-18T11:54:51.303644+0000","last_undegraded":"2024-09-18T11:54:51.303644+0000","last_fullsized":"2024-09-18T11:54:51.303644+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11639204,"num_objects":2845,"num_object_clones":0,"num_object_copies":5690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2850,"num_write_kb":11371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":192130730,"num_objects":46929,"num_object_clones":0,"num_object_copies":93858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44152,"num_write_kb":176446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":386093056,"data_stored":385988686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46985,"ondisk_log_size":46985,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051596,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485164,"kb_used_data":187300,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93886676,"statfs":{"total":96636764160,"available":96139956224,"internally_reserved":0,"allocated":191795200,"data_stored":191611918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19800000000000001}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478428,"kb_used_data":180628,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93893412,"statfs":{"total":96636764160,"available":96146853888,"internally_reserved":0,"allocated":184963072,"data_stored":184779790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,3,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":189792256,"data_stored":189788777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":196186112,"data_stored":196182633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:54:53.264 INFO:tasks.rep_lost_unfound_delete.ceph_manager:recovered! 2024-09-18T11:54:53.265 INFO:tasks.util.rados:rados -p unfounddel_pool get new_1 - 2024-09-18T11:54:53.265 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_1 - 2024-09-18T11:54:53.339 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/new_1: (2) No such file or directory 2024-09-18T11:54:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:53.350 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_1 - 2024-09-18T11:54:53.350 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_1 - 2024-09-18T11:54:53.407 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existed_1: (2) No such file or directory 2024-09-18T11:54:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:53.410 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_1 - 2024-09-18T11:54:53.410 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_1 - 2024-09-18T11:54:53.474 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existing_1: (2) No such file or directory 2024-09-18T11:54:53.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:53.481 INFO:tasks.util.rados:rados -p unfounddel_pool get new_2 - 2024-09-18T11:54:53.481 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_2 - 2024-09-18T11:54:53.555 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/new_2: (2) No such file or directory 2024-09-18T11:54:53.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:53.561 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_2 - 2024-09-18T11:54:53.561 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_2 - 2024-09-18T11:54:53.618 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existed_2: (2) No such file or directory 2024-09-18T11:54:53.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:53.627 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_2 - 2024-09-18T11:54:53.627 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_2 - 2024-09-18T11:54:53.685 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existing_2: (2) No such file or directory 2024-09-18T11:54:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:53.707 INFO:tasks.util.rados:rados -p unfounddel_pool get new_3 - 2024-09-18T11:54:53.707 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_3 - 2024-09-18T11:54:53.787 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/new_3: (2) No such file or directory 2024-09-18T11:54:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:53.803 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_3 - 2024-09-18T11:54:53.804 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_3 - 2024-09-18T11:54:53.868 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existed_3: (2) No such file or directory 2024-09-18T11:54:53.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:53.890 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_3 - 2024-09-18T11:54:53.890 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_3 - 2024-09-18T11:54:53.966 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existing_3: (2) No such file or directory 2024-09-18T11:54:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:53.974 INFO:tasks.util.rados:rados -p unfounddel_pool get new_4 - 2024-09-18T11:54:53.974 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_4 - 2024-09-18T11:54:54.043 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/new_4: (2) No such file or directory 2024-09-18T11:54:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.055 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_4 - 2024-09-18T11:54:54.055 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_4 - 2024-09-18T11:54:54.097 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 28 20 49942 49922 6.96328 4.72266 0.0122147 0.0112143 2024-09-18T11:54:54.134 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existed_4: (2) No such file or directory 2024-09-18T11:54:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.140 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_4 - 2024-09-18T11:54:54.141 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_4 - 2024-09-18T11:54:54.204 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existing_4: (2) No such file or directory 2024-09-18T11:54:54.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.217 INFO:tasks.util.rados:rados -p unfounddel_pool get new_5 - 2024-09-18T11:54:54.217 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_5 - 2024-09-18T11:54:54.282 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/new_5: (2) No such file or directory 2024-09-18T11:54:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.299 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_5 - 2024-09-18T11:54:54.299 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_5 - 2024-09-18T11:54:54.352 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existed_5: (2) No such file or directory 2024-09-18T11:54:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.354 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_5 - 2024-09-18T11:54:54.354 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_5 - 2024-09-18T11:54:54.429 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existing_5: (2) No such file or directory 2024-09-18T11:54:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.438 INFO:tasks.util.rados:rados -p unfounddel_pool get new_6 - 2024-09-18T11:54:54.438 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_6 - 2024-09-18T11:54:54.501 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/new_6: (2) No such file or directory 2024-09-18T11:54:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.508 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_6 - 2024-09-18T11:54:54.508 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_6 - 2024-09-18T11:54:54.571 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existed_6: (2) No such file or directory 2024-09-18T11:54:54.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.585 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_6 - 2024-09-18T11:54:54.585 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_6 - 2024-09-18T11:54:54.648 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existing_6: (2) No such file or directory 2024-09-18T11:54:54.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.663 INFO:tasks.util.rados:rados -p unfounddel_pool get new_7 - 2024-09-18T11:54:54.663 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_7 - 2024-09-18T11:54:54.708 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/new_7: (2) No such file or directory 2024-09-18T11:54:54.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.720 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_7 - 2024-09-18T11:54:54.721 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_7 - 2024-09-18T11:54:54.836 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existed_7: (2) No such file or directory 2024-09-18T11:54:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.841 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_7 - 2024-09-18T11:54:54.841 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_7 - 2024-09-18T11:54:54.903 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existing_7: (2) No such file or directory 2024-09-18T11:54:54.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.908 INFO:tasks.util.rados:rados -p unfounddel_pool get new_8 - 2024-09-18T11:54:54.908 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_8 - 2024-09-18T11:54:54.982 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/new_8: (2) No such file or directory 2024-09-18T11:54:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:54.993 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_8 - 2024-09-18T11:54:54.993 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_8 - 2024-09-18T11:54:55.072 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existed_8: (2) No such file or directory 2024-09-18T11:54:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:55.076 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_8 - 2024-09-18T11:54:55.076 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_8 - 2024-09-18T11:54:55.097 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 29 20 51086 51066 6.87726 4.46875 0.0134048 0.0113526 2024-09-18T11:54:55.137 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existing_8: (2) No such file or directory 2024-09-18T11:54:55.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:55.151 INFO:tasks.util.rados:rados -p unfounddel_pool get new_9 - 2024-09-18T11:54:55.151 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_9 - 2024-09-18T11:54:55.276 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/new_9: (2) No such file or directory 2024-09-18T11:54:55.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:55.279 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_9 - 2024-09-18T11:54:55.279 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_9 - 2024-09-18T11:54:55.355 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existed_9: (2) No such file or directory 2024-09-18T11:54:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:55.369 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_9 - 2024-09-18T11:54:55.369 DEBUG:teuthology.orchestra.run.smithi154:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_9 - 2024-09-18T11:54:55.430 INFO:teuthology.orchestra.run.smithi154.stderr:error getting unfounddel_pool/existing_9: (2) No such file or directory 2024-09-18T11:54:55.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:54:55.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-09-18T11:54:55.886 INFO:tasks.ceph.mon.a.smithi154.stderr:2024-09-18T11:54:55.880+0000 7f2c7dcba700 -1 mon.a@0(leader).osd e30 definitely_dead 0 2024-09-18T11:54:56.097 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 30 20 52223 52203 6.79606 4.44141 0.00856433 0.0114889 2024-09-18T11:54:56.179 INFO:teuthology.orchestra.run.smithi154.stderr:marked in osd.1. 2024-09-18T11:54:56.192 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-18T11:54:56.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-18T11:54:56.194 INFO:tasks.ceph.osd.1:Started 2024-09-18T11:54:56.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T11:54:56.286 INFO:teuthology.orchestra.run.smithi154.stderr:no valid command found; 10 closest matches: 2024-09-18T11:54:56.286 INFO:teuthology.orchestra.run.smithi154.stderr:0 2024-09-18T11:54:56.286 INFO:teuthology.orchestra.run.smithi154.stderr:1 2024-09-18T11:54:56.286 INFO:teuthology.orchestra.run.smithi154.stderr:2 2024-09-18T11:54:56.286 INFO:teuthology.orchestra.run.smithi154.stderr:abort 2024-09-18T11:54:56.287 INFO:teuthology.orchestra.run.smithi154.stderr:assert 2024-09-18T11:54:56.287 INFO:teuthology.orchestra.run.smithi154.stderr:config diff 2024-09-18T11:54:56.287 INFO:teuthology.orchestra.run.smithi154.stderr:config diff get 2024-09-18T11:54:56.287 INFO:teuthology.orchestra.run.smithi154.stderr:config get 2024-09-18T11:54:56.287 INFO:teuthology.orchestra.run.smithi154.stderr:config help [] 2024-09-18T11:54:56.287 INFO:teuthology.orchestra.run.smithi154.stderr:config set ... 2024-09-18T11:54:56.287 INFO:teuthology.orchestra.run.smithi154.stderr:admin_socket: invalid command 2024-09-18T11:54:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T11:54:56.288 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-18T11:54:56.797 INFO:tasks.ceph.osd.1.smithi154.stderr:2024-09-18T11:54:56.792+0000 7fc932908540 -1 Falling back to public interface 2024-09-18T11:54:57.097 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 31 20 52325 52305 6.58969 0.398438 0.0296787 0.0115017 2024-09-18T11:54:58.097 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 32 20 52325 52305 6.38377 0 - 0.0115017 2024-09-18T11:54:59.098 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 33 20 53363 53343 6.31314 2.02734 0.0503945 0.0123662 2024-09-18T11:54:59.152 INFO:tasks.ceph.osd.1.smithi154.stderr:2024-09-18T11:54:59.148+0000 7fc932908540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T11:54:59.964 INFO:tasks.ceph.osd.1.smithi154.stderr:2024-09-18T11:54:59.956+0000 7fc932908540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T11:55:00.098 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 34 20 54483 54463 6.25613 4.375 0.00470001 0.0124814 2024-09-18T11:55:00.373 INFO:tasks.ceph.osd.1.smithi154.stderr:2024-09-18T11:55:00.368+0000 7fc932908540 -1 osd.1 24 log_to_monitors true 2024-09-18T11:55:01.098 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 35 20 55629 55609 6.20528 4.47656 0.0126503 0.0125836 2024-09-18T11:55:01.237 INFO:tasks.ceph.osd.1.smithi154.stderr:2024-09-18T11:55:01.232+0000 7fc92764c700 -1 osd.1 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T11:55:01.289 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T11:55:01.433 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T11:55:01.433 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-18T11:55:01.433 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-18T11:55:01.433 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T11:55:01.445 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for osd.1 to be up 2024-09-18T11:55:01.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:55:01.889 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:01.889 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":34,"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","created":"2024-09-18T11:53:20.028520+0000","modified":"2024-09-18T11:55:01.210075+0000","last_up_change":"2024-09-18T11:54:17.405238+0000","last_in_change":"2024-09-18T11:54:56.163654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:53:29.033564+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-09-18T11:53:40.807258+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":19,"up_from":23,"up_thru":32,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":26420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":26420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":26420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":26420}]},"public_addr":"172.21.15.154:6800/26420","cluster_addr":"172.21.15.154:6801/26420","heartbeat_back_addr":"172.21.15.154:6803/26420","heartbeat_front_addr":"172.21.15.154:6802/26420","state":["exists","up"]},{"osd":1,"uuid":"fd6ca4f9-93c5-4509-8fea-0121afa900f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":23,"down_at":25,"lost_at":25,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":21134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":21134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":21134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":21134}]},"public_addr":"172.21.15.154:6804/21134","cluster_addr":"172.21.15.154:6805/21134","heartbeat_back_addr":"172.21.15.154:6807/21134","heartbeat_front_addr":"172.21.15.154:6806/21134","state":["exists"]},{"osd":2,"uuid":"1554e072-019d-4d8e-b2ee-c0aac5a5e811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":15,"up_from":29,"up_thru":32,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":27666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":27666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":27666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":27666}]},"public_addr":"172.21.15.154:6804/27666","cluster_addr":"172.21.15.154:6805/27666","heartbeat_back_addr":"172.21.15.154:6807/27666","heartbeat_front_addr":"172.21.15.154:6806/27666","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.685684+0000","dead_epoch":20},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.545391+0000","dead_epoch":25},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.679601+0000","dead_epoch":17}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[0,2]},{"pgid":"1.2","osds":[0,2]},{"pgid":"1.3","osds":[2,0]},{"pgid":"1.4","osds":[0,2]},{"pgid":"1.6","osds":[0,2]},{"pgid":"1.7","osds":[2,0]},{"pgid":"2.0","osds":[2,0]},{"pgid":"2.1","osds":[2,0]},{"pgid":"2.2","osds":[0,2]},{"pgid":"2.3","osds":[2,0]},{"pgid":"2.4","osds":[0,2]},{"pgid":"2.5","osds":[0,2]},{"pgid":"2.6","osds":[0,2]},{"pgid":"2.7","osds":[0,2]},{"pgid":"2.9","osds":[0,2]},{"pgid":"2.a","osds":[0,2]},{"pgid":"2.d","osds":[2,0]},{"pgid":"2.f","osds":[0,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:55:02.098 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 36 20 56679 56659 6.14684 4.10156 0.0125114 0.0127046 2024-09-18T11:55:03.098 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 37 20 56836 56816 5.99729 0.613281 0.0031527 0.0127196 2024-09-18T11:55:04.098 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 38 20 56836 56816 5.83947 0 - 0.0127196 2024-09-18T11:55:04.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T11:55:05.098 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 39 20 57044 57024 5.71059 0.40625 0.0219943 0.0136686 2024-09-18T11:55:05.627 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:05.627 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":37,"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","created":"2024-09-18T11:53:20.028520+0000","modified":"2024-09-18T11:55:04.237643+0000","last_up_change":"2024-09-18T11:55:02.220717+0000","last_in_change":"2024-09-18T11:54:56.163654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:53:29.033564+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-09-18T11:53:40.807258+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":19,"up_from":23,"up_thru":36,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":26420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":26420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":26420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":26420}]},"public_addr":"172.21.15.154:6800/26420","cluster_addr":"172.21.15.154:6801/26420","heartbeat_back_addr":"172.21.15.154:6803/26420","heartbeat_front_addr":"172.21.15.154:6802/26420","state":["exists","up"]},{"osd":1,"uuid":"fd6ca4f9-93c5-4509-8fea-0121afa900f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":24,"up_from":35,"up_thru":36,"down_at":25,"lost_at":25,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":30839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":30839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":30839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":30839}]},"public_addr":"172.21.15.154:6808/30839","cluster_addr":"172.21.15.154:6809/30839","heartbeat_back_addr":"172.21.15.154:6811/30839","heartbeat_front_addr":"172.21.15.154:6810/30839","state":["exists","up"]},{"osd":2,"uuid":"1554e072-019d-4d8e-b2ee-c0aac5a5e811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":15,"up_from":29,"up_thru":36,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":27666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":27666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":27666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":27666}]},"public_addr":"172.21.15.154:6804/27666","cluster_addr":"172.21.15.154:6805/27666","heartbeat_back_addr":"172.21.15.154:6807/27666","heartbeat_front_addr":"172.21.15.154:6806/27666","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.685684+0000","dead_epoch":20},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.545391+0000","dead_epoch":25},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.679601+0000","dead_epoch":17}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"2.0","osds":[2]},{"pgid":"2.1","osds":[2]},{"pgid":"2.2","osds":[0]},{"pgid":"2.3","osds":[2]},{"pgid":"2.4","osds":[0]},{"pgid":"2.5","osds":[0]},{"pgid":"2.6","osds":[0]},{"pgid":"2.7","osds":[0]},{"pgid":"2.9","osds":[0]},{"pgid":"2.a","osds":[0]},{"pgid":"2.d","osds":[2]},{"pgid":"2.f","osds":[0]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T11:55:05.656 INFO:tasks.rep_lost_unfound_delete.ceph_manager:osd.1 is up 2024-09-18T11:55:05.656 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for clean 2024-09-18T11:55:05.656 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:06.098 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:2024-09-18T11:55:06.098749+0000 min lat: 0.0021114 max lat: 2.57663 avg lat: 0.0138778 2024-09-18T11:55:06.098 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:55:06.099 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 40 20 57637 57617 5.62573 2.31641 0.0441887 0.0138778 2024-09-18T11:55:06.253 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:06.281 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:06.321 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-18T11:55:05.039618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":230302384,"num_objects":56246,"num_object_clones":0,"num_object_copies":112492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27933,"num_objects_unfound":0,"num_objects_dirty":56246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43982,"num_write_kb":175799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56308,"ondisk_log_size":56308,"up":36,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1343320,"kb_used_data":449784,"kb_used_omap":13,"kb_used_meta":893490,"kb_avail":281772200,"statfs":{"total":289910292480,"available":288534732800,"internally_reserved":0,"allocated":460578816,"data_stored":459928384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":914934290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,9,3,0,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15993646,"num_objects":3903,"num_object_clones":0,"num_object_copies":7806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27933,"num_objects_unfound":0,"num_objects_dirty":3903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":387,"num_write_kb":1561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001320"},"pg_stats":[{"pgid":"2.f","version":"34'3539","reported_seq":3596,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.268275+0000","last_change":"2024-09-18T11:54:58.223970+0000","last_active":"2024-09-18T11:55:01.268275+0000","last_peered":"2024-09-18T11:55:01.268275+0000","last_clean":"2024-09-18T11:55:01.268275+0000","last_became_active":"2024-09-18T11:54:58.217048+0000","last_became_peered":"2024-09-18T11:54:58.217048+0000","last_unstale":"2024-09-18T11:55:01.268275+0000","last_undegraded":"2024-09-18T11:55:01.268275+0000","last_fullsized":"2024-09-18T11:55:01.268275+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3539,"log_dups_size":0,"ondisk_log_size":3539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:42:42.307745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14495744,"num_objects":3539,"num_object_clones":0,"num_object_copies":7078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3539,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":14156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3539}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"34'3558","reported_seq":3712,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:55:02.246157+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:02.246157+0000","last_peered":"2024-09-18T11:55:02.246157+0000","last_clean":"2024-09-18T11:55:02.246157+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:02.246157+0000","last_undegraded":"2024-09-18T11:55:02.246157+0000","last_fullsized":"2024-09-18T11:55:02.246157+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14548992,"num_objects":3552,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":750,"num_write_kb":3000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"34'3540","reported_seq":3713,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.246687+0000","last_change":"2024-09-18T11:55:02.249367+0000","last_active":"2024-09-18T11:55:02.246398+0000","last_peered":"2024-09-18T11:55:02.215905+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:54:58.199903+0000","last_became_peered":"2024-09-18T11:54:58.199903+0000","last_unstale":"2024-09-18T11:55:02.246687+0000","last_undegraded":"2024-09-18T11:55:02.246687+0000","last_fullsized":"2024-09-18T11:55:02.246687+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":3540,"log_dups_size":0,"ondisk_log_size":3540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:26:33.136656+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14483456,"num_objects":3536,"num_object_clones":0,"num_object_copies":7072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":398,"num_write_kb":1592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"34'3503","reported_seq":3528,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:55:02.240529+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:02.240529+0000","last_peered":"2024-09-18T11:55:02.240529+0000","last_clean":"2024-09-18T11:55:02.240529+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:02.240529+0000","last_undegraded":"2024-09-18T11:55:02.240529+0000","last_fullsized":"2024-09-18T11:55:02.240529+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14348288,"num_objects":3503,"num_object_clones":0,"num_object_copies":7006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":14012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"34'3486","reported_seq":3672,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.308027+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:01.308027+0000","last_peered":"2024-09-18T11:55:01.308027+0000","last_clean":"2024-09-18T11:55:01.308027+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:01.308027+0000","last_undegraded":"2024-09-18T11:55:01.308027+0000","last_fullsized":"2024-09-18T11:55:01.308027+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":3486,"log_dups_size":0,"ondisk_log_size":3486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:49:10.796724+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14249984,"num_objects":3479,"num_object_clones":0,"num_object_copies":6958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":651,"num_write_kb":2604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"34'3472","reported_seq":3540,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.336054+0000","last_change":"2024-09-18T11:54:58.213851+0000","last_active":"2024-09-18T11:55:01.336054+0000","last_peered":"2024-09-18T11:55:01.336054+0000","last_clean":"2024-09-18T11:55:01.336054+0000","last_became_active":"2024-09-18T11:54:58.208493+0000","last_became_peered":"2024-09-18T11:54:58.208493+0000","last_unstale":"2024-09-18T11:55:01.336054+0000","last_undegraded":"2024-09-18T11:55:01.336054+0000","last_fullsized":"2024-09-18T11:55:01.336054+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3472,"log_dups_size":0,"ondisk_log_size":3472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14202066,"num_objects":3469,"num_object_clones":0,"num_object_copies":6938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3469,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3470,"num_write_kb":13871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3469}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"34'3508","reported_seq":3581,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.336468+0000","last_change":"2024-09-18T11:54:58.251037+0000","last_active":"2024-09-18T11:55:01.336468+0000","last_peered":"2024-09-18T11:55:01.336468+0000","last_clean":"2024-09-18T11:55:01.336468+0000","last_became_active":"2024-09-18T11:54:58.233086+0000","last_became_peered":"2024-09-18T11:54:58.233086+0000","last_unstale":"2024-09-18T11:55:01.336468+0000","last_undegraded":"2024-09-18T11:55:01.336468+0000","last_fullsized":"2024-09-18T11:55:01.336468+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3508,"log_dups_size":0,"ondisk_log_size":3508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14345426,"num_objects":3504,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3504,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3506,"num_write_kb":14012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3504}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"34'3617","reported_seq":3680,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.280169+0000","last_change":"2024-09-18T11:54:47.832713+0000","last_active":"2024-09-18T11:55:01.280169+0000","last_peered":"2024-09-18T11:55:01.280169+0000","last_clean":"2024-09-18T11:55:01.280169+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:01.280169+0000","last_undegraded":"2024-09-18T11:55:01.280169+0000","last_fullsized":"2024-09-18T11:55:01.280169+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3617,"log_dups_size":0,"ondisk_log_size":3617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14807657,"num_objects":3616,"num_object_clones":0,"num_object_copies":7232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3616,"num_write_kb":14461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":33,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:54:58.209953+0000","last_change":"2024-09-18T11:54:58.199411+0000","last_active":"2024-09-18T11:54:58.209953+0000","last_peered":"2024-09-18T11:54:58.209953+0000","last_clean":"2024-09-18T11:54:58.209953+0000","last_became_active":"2024-09-18T11:54:58.198635+0000","last_became_peered":"2024-09-18T11:54:58.198635+0000","last_unstale":"2024-09-18T11:54:58.209953+0000","last_undegraded":"2024-09-18T11:54:58.209953+0000","last_fullsized":"2024-09-18T11:54:58.209953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:58:12.597223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"34'3486","reported_seq":3627,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.247293+0000","last_change":"2024-09-18T11:55:02.249595+0000","last_active":"2024-09-18T11:55:02.247234+0000","last_peered":"2024-09-18T11:55:02.234203+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:54:58.199897+0000","last_became_peered":"2024-09-18T11:54:58.199897+0000","last_unstale":"2024-09-18T11:55:02.247293+0000","last_undegraded":"2024-09-18T11:55:02.247293+0000","last_fullsized":"2024-09-18T11:55:02.247293+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":3486,"log_dups_size":0,"ondisk_log_size":3486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14251218,"num_objects":3481,"num_object_clones":0,"num_object_copies":6962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":13920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.237200+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:01.237200+0000","last_peered":"2024-09-18T11:55:01.237200+0000","last_clean":"2024-09-18T11:55:01.237200+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:01.237200+0000","last_undegraded":"2024-09-18T11:55:01.237200+0000","last_fullsized":"2024-09-18T11:55:01.237200+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:14:25.540910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"34'3491","reported_seq":3554,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.297659+0000","last_change":"2024-09-18T11:54:58.212475+0000","last_active":"2024-09-18T11:55:01.297659+0000","last_peered":"2024-09-18T11:55:01.297659+0000","last_clean":"2024-09-18T11:55:01.297659+0000","last_became_active":"2024-09-18T11:54:58.207790+0000","last_became_peered":"2024-09-18T11:54:58.207790+0000","last_unstale":"2024-09-18T11:55:01.297659+0000","last_undegraded":"2024-09-18T11:55:01.297659+0000","last_fullsized":"2024-09-18T11:55:01.297659+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14267602,"num_objects":3485,"num_object_clones":0,"num_object_copies":6970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3485,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":13936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":75,"reported_epoch":33,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:54:58.213412+0000","last_change":"2024-09-18T11:54:58.212094+0000","last_active":"2024-09-18T11:54:58.213412+0000","last_peered":"2024-09-18T11:54:58.213412+0000","last_clean":"2024-09-18T11:54:58.213412+0000","last_became_active":"2024-09-18T11:54:58.207631+0000","last_became_peered":"2024-09-18T11:54:58.207631+0000","last_unstale":"2024-09-18T11:54:58.213412+0000","last_undegraded":"2024-09-18T11:54:58.213412+0000","last_fullsized":"2024-09-18T11:54:58.213412+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:53:31.760701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"34'3555","reported_seq":3599,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.241047+0000","last_change":"2024-09-18T11:55:02.241598+0000","last_active":"2024-09-18T11:55:02.241015+0000","last_peered":"2024-09-18T11:55:02.235429+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:54:58.219699+0000","last_became_peered":"2024-09-18T11:54:58.219699+0000","last_unstale":"2024-09-18T11:55:02.241047+0000","last_undegraded":"2024-09-18T11:55:02.241047+0000","last_fullsized":"2024-09-18T11:55:02.241047+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3555,"log_dups_size":0,"ondisk_log_size":3555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14542034,"num_objects":3552,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3553,"num_write_kb":14203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":39,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.251896+0000","last_change":"2024-09-18T11:55:02.252085+0000","last_active":"2024-09-18T11:55:02.251871+0000","last_peered":"2024-09-18T11:55:01.240094+0000","last_clean":"2024-09-18T11:55:01.240094+0000","last_became_active":"2024-09-18T11:54:58.214590+0000","last_became_peered":"2024-09-18T11:54:58.214590+0000","last_unstale":"2024-09-18T11:55:02.251896+0000","last_undegraded":"2024-09-18T11:55:02.251896+0000","last_fullsized":"2024-09-18T11:55:02.251896+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:48.512243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"34'3599","reported_seq":3796,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.243113+0000","last_change":"2024-09-18T11:55:02.247155+0000","last_active":"2024-09-18T11:55:02.243084+0000","last_peered":"2024-09-18T11:55:02.228397+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:54:58.203629+0000","last_became_peered":"2024-09-18T11:54:58.203629+0000","last_unstale":"2024-09-18T11:55:02.243113+0000","last_undegraded":"2024-09-18T11:55:02.243113+0000","last_fullsized":"2024-09-18T11:55:02.243113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:09:29.956333+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14716928,"num_objects":3593,"num_object_clones":0,"num_object_copies":7186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"34'3475","reported_seq":3543,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.309320+0000","last_change":"2024-09-18T11:54:58.268664+0000","last_active":"2024-09-18T11:55:01.309320+0000","last_peered":"2024-09-18T11:55:01.309320+0000","last_clean":"2024-09-18T11:55:01.309320+0000","last_became_active":"2024-09-18T11:54:58.244264+0000","last_became_peered":"2024-09-18T11:54:58.244264+0000","last_unstale":"2024-09-18T11:55:01.309320+0000","last_undegraded":"2024-09-18T11:55:01.309320+0000","last_fullsized":"2024-09-18T11:55:01.309320+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3475,"log_dups_size":0,"ondisk_log_size":3475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14226025,"num_objects":3474,"num_object_clones":0,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3474,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3474,"num_write_kb":13893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3474}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":33,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:54:58.221287+0000","last_change":"2024-09-18T11:54:58.214786+0000","last_active":"2024-09-18T11:54:58.221287+0000","last_peered":"2024-09-18T11:54:58.221287+0000","last_clean":"2024-09-18T11:54:58.221287+0000","last_became_active":"2024-09-18T11:54:58.214449+0000","last_became_peered":"2024-09-18T11:54:58.214449+0000","last_unstale":"2024-09-18T11:54:58.221287+0000","last_undegraded":"2024-09-18T11:54:58.221287+0000","last_fullsized":"2024-09-18T11:54:58.221287+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:22:21.910099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"34'3472","reported_seq":3542,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.311932+0000","last_change":"2024-09-18T11:54:58.216748+0000","last_active":"2024-09-18T11:55:01.311932+0000","last_peered":"2024-09-18T11:55:01.311932+0000","last_clean":"2024-09-18T11:55:01.311932+0000","last_became_active":"2024-09-18T11:54:58.209450+0000","last_became_peered":"2024-09-18T11:54:58.209450+0000","last_unstale":"2024-09-18T11:55:01.311932+0000","last_undegraded":"2024-09-18T11:55:01.311932+0000","last_fullsized":"2024-09-18T11:55:01.311932+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3472,"log_dups_size":0,"ondisk_log_size":3472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14190395,"num_objects":3467,"num_object_clones":0,"num_object_copies":6934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3467,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3469,"num_write_kb":13860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3467}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.242328+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:01.242328+0000","last_peered":"2024-09-18T11:55:01.242328+0000","last_clean":"2024-09-18T11:55:01.242328+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:01.242328+0000","last_undegraded":"2024-09-18T11:55:01.242328+0000","last_fullsized":"2024-09-18T11:55:01.242328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:10:36.497525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"34'3534","reported_seq":3601,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.321683+0000","last_change":"2024-09-18T11:54:58.213368+0000","last_active":"2024-09-18T11:55:01.321683+0000","last_peered":"2024-09-18T11:55:01.321683+0000","last_clean":"2024-09-18T11:55:01.321683+0000","last_became_active":"2024-09-18T11:54:58.208450+0000","last_became_peered":"2024-09-18T11:54:58.208450+0000","last_unstale":"2024-09-18T11:55:01.321683+0000","last_undegraded":"2024-09-18T11:55:01.321683+0000","last_fullsized":"2024-09-18T11:55:01.321683+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14460114,"num_objects":3532,"num_object_clones":0,"num_object_copies":7064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3532,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3532,"num_write_kb":14122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":33,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:54:58.223862+0000","last_change":"2024-09-18T11:54:58.215175+0000","last_active":"2024-09-18T11:54:58.223862+0000","last_peered":"2024-09-18T11:54:58.223862+0000","last_clean":"2024-09-18T11:54:58.223862+0000","last_became_active":"2024-09-18T11:54:58.207671+0000","last_became_peered":"2024-09-18T11:54:58.207671+0000","last_unstale":"2024-09-18T11:54:58.223862+0000","last_undegraded":"2024-09-18T11:54:58.223862+0000","last_fullsized":"2024-09-18T11:54:58.223862+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:45:00.297452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.246125+0000","last_change":"2024-09-18T11:55:02.246320+0000","last_active":"2024-09-18T11:55:02.244640+0000","last_peered":"2024-09-18T11:55:01.242430+0000","last_clean":"2024-09-18T11:55:01.242430+0000","last_became_active":"2024-09-18T11:54:58.229836+0000","last_became_peered":"2024-09-18T11:54:58.229836+0000","last_unstale":"2024-09-18T11:55:02.246125+0000","last_undegraded":"2024-09-18T11:55:02.246125+0000","last_fullsized":"2024-09-18T11:55:02.246125+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:28:51.494138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"34'3470","reported_seq":3541,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.324185+0000","last_change":"2024-09-18T11:54:58.212484+0000","last_active":"2024-09-18T11:55:01.324185+0000","last_peered":"2024-09-18T11:55:01.324185+0000","last_clean":"2024-09-18T11:55:01.324185+0000","last_became_active":"2024-09-18T11:54:58.207705+0000","last_became_peered":"2024-09-18T11:54:58.207705+0000","last_unstale":"2024-09-18T11:55:01.324185+0000","last_undegraded":"2024-09-18T11:55:01.324185+0000","last_fullsized":"2024-09-18T11:55:01.324185+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14166436,"num_objects":3462,"num_object_clones":0,"num_object_copies":6924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3462,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3467,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3462}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":230302365,"num_objects":56244,"num_object_clones":0,"num_object_copies":112488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27932,"num_objects_unfound":0,"num_objects_dirty":56244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43980,"num_write_kb":175797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":461791232,"data_stored":461686862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56305,"ondisk_log_size":56305,"up":24,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051598,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523532,"kb_used_data":225664,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93848308,"statfs":{"total":96636764160,"available":96100667392,"internally_reserved":0,"allocated":231079936,"data_stored":230877841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298180,"kb_used_data":316,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073660,"statfs":{"total":96636764160,"available":96331427840,"internally_reserved":0,"allocated":323584,"data_stored":77342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521608,"kb_used_data":223804,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93850232,"statfs":{"total":96636764160,"available":96102637568,"internally_reserved":0,"allocated":229175296,"data_stored":228973201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,9,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":228917248,"data_stored":228913769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":232759296,"data_stored":232755817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:06.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:06.976 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:06.983 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:07.001 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-18T11:55:05.039618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":230302384,"num_objects":56246,"num_object_clones":0,"num_object_copies":112492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27933,"num_objects_unfound":0,"num_objects_dirty":56246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43982,"num_write_kb":175799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":8828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56308,"ondisk_log_size":56308,"up":36,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1343320,"kb_used_data":449784,"kb_used_omap":13,"kb_used_meta":893490,"kb_avail":281772200,"statfs":{"total":289910292480,"available":288534732800,"internally_reserved":0,"allocated":460578816,"data_stored":459928384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":914934290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,9,3,0,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15993646,"num_objects":3903,"num_object_clones":0,"num_object_copies":7806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27933,"num_objects_unfound":0,"num_objects_dirty":3903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":387,"num_write_kb":1561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001320"},"pg_stats":[{"pgid":"2.f","version":"34'3539","reported_seq":3596,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.268275+0000","last_change":"2024-09-18T11:54:58.223970+0000","last_active":"2024-09-18T11:55:01.268275+0000","last_peered":"2024-09-18T11:55:01.268275+0000","last_clean":"2024-09-18T11:55:01.268275+0000","last_became_active":"2024-09-18T11:54:58.217048+0000","last_became_peered":"2024-09-18T11:54:58.217048+0000","last_unstale":"2024-09-18T11:55:01.268275+0000","last_undegraded":"2024-09-18T11:55:01.268275+0000","last_fullsized":"2024-09-18T11:55:01.268275+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3539,"log_dups_size":0,"ondisk_log_size":3539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:42:42.307745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14495744,"num_objects":3539,"num_object_clones":0,"num_object_copies":7078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3539,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":14156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3539}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"34'3558","reported_seq":3712,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:55:02.246157+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:02.246157+0000","last_peered":"2024-09-18T11:55:02.246157+0000","last_clean":"2024-09-18T11:55:02.246157+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:02.246157+0000","last_undegraded":"2024-09-18T11:55:02.246157+0000","last_fullsized":"2024-09-18T11:55:02.246157+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14548992,"num_objects":3552,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":750,"num_write_kb":3000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"34'3540","reported_seq":3713,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.246687+0000","last_change":"2024-09-18T11:55:02.249367+0000","last_active":"2024-09-18T11:55:02.246398+0000","last_peered":"2024-09-18T11:55:02.215905+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:54:58.199903+0000","last_became_peered":"2024-09-18T11:54:58.199903+0000","last_unstale":"2024-09-18T11:55:02.246687+0000","last_undegraded":"2024-09-18T11:55:02.246687+0000","last_fullsized":"2024-09-18T11:55:02.246687+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":3540,"log_dups_size":0,"ondisk_log_size":3540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:26:33.136656+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14483456,"num_objects":3536,"num_object_clones":0,"num_object_copies":7072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":398,"num_write_kb":1592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"34'3503","reported_seq":3528,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:55:02.240529+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:02.240529+0000","last_peered":"2024-09-18T11:55:02.240529+0000","last_clean":"2024-09-18T11:55:02.240529+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:02.240529+0000","last_undegraded":"2024-09-18T11:55:02.240529+0000","last_fullsized":"2024-09-18T11:55:02.240529+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14348288,"num_objects":3503,"num_object_clones":0,"num_object_copies":7006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":14012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"34'3486","reported_seq":3672,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.308027+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:01.308027+0000","last_peered":"2024-09-18T11:55:01.308027+0000","last_clean":"2024-09-18T11:55:01.308027+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:01.308027+0000","last_undegraded":"2024-09-18T11:55:01.308027+0000","last_fullsized":"2024-09-18T11:55:01.308027+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":3486,"log_dups_size":0,"ondisk_log_size":3486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:49:10.796724+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14249984,"num_objects":3479,"num_object_clones":0,"num_object_copies":6958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":651,"num_write_kb":2604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"34'3472","reported_seq":3540,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.336054+0000","last_change":"2024-09-18T11:54:58.213851+0000","last_active":"2024-09-18T11:55:01.336054+0000","last_peered":"2024-09-18T11:55:01.336054+0000","last_clean":"2024-09-18T11:55:01.336054+0000","last_became_active":"2024-09-18T11:54:58.208493+0000","last_became_peered":"2024-09-18T11:54:58.208493+0000","last_unstale":"2024-09-18T11:55:01.336054+0000","last_undegraded":"2024-09-18T11:55:01.336054+0000","last_fullsized":"2024-09-18T11:55:01.336054+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3472,"log_dups_size":0,"ondisk_log_size":3472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14202066,"num_objects":3469,"num_object_clones":0,"num_object_copies":6938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3469,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3470,"num_write_kb":13871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3469}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"34'3508","reported_seq":3581,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.336468+0000","last_change":"2024-09-18T11:54:58.251037+0000","last_active":"2024-09-18T11:55:01.336468+0000","last_peered":"2024-09-18T11:55:01.336468+0000","last_clean":"2024-09-18T11:55:01.336468+0000","last_became_active":"2024-09-18T11:54:58.233086+0000","last_became_peered":"2024-09-18T11:54:58.233086+0000","last_unstale":"2024-09-18T11:55:01.336468+0000","last_undegraded":"2024-09-18T11:55:01.336468+0000","last_fullsized":"2024-09-18T11:55:01.336468+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3508,"log_dups_size":0,"ondisk_log_size":3508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14345426,"num_objects":3504,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3504,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3506,"num_write_kb":14012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3504}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"34'3617","reported_seq":3680,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.280169+0000","last_change":"2024-09-18T11:54:47.832713+0000","last_active":"2024-09-18T11:55:01.280169+0000","last_peered":"2024-09-18T11:55:01.280169+0000","last_clean":"2024-09-18T11:55:01.280169+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:01.280169+0000","last_undegraded":"2024-09-18T11:55:01.280169+0000","last_fullsized":"2024-09-18T11:55:01.280169+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3617,"log_dups_size":0,"ondisk_log_size":3617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14807657,"num_objects":3616,"num_object_clones":0,"num_object_copies":7232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3616,"num_write_kb":14461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":33,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:54:58.209953+0000","last_change":"2024-09-18T11:54:58.199411+0000","last_active":"2024-09-18T11:54:58.209953+0000","last_peered":"2024-09-18T11:54:58.209953+0000","last_clean":"2024-09-18T11:54:58.209953+0000","last_became_active":"2024-09-18T11:54:58.198635+0000","last_became_peered":"2024-09-18T11:54:58.198635+0000","last_unstale":"2024-09-18T11:54:58.209953+0000","last_undegraded":"2024-09-18T11:54:58.209953+0000","last_fullsized":"2024-09-18T11:54:58.209953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:58:12.597223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"34'3486","reported_seq":3627,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.247293+0000","last_change":"2024-09-18T11:55:02.249595+0000","last_active":"2024-09-18T11:55:02.247234+0000","last_peered":"2024-09-18T11:55:02.234203+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:54:58.199897+0000","last_became_peered":"2024-09-18T11:54:58.199897+0000","last_unstale":"2024-09-18T11:55:02.247293+0000","last_undegraded":"2024-09-18T11:55:02.247293+0000","last_fullsized":"2024-09-18T11:55:02.247293+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":3486,"log_dups_size":0,"ondisk_log_size":3486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14251218,"num_objects":3481,"num_object_clones":0,"num_object_copies":6962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":13920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.237200+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:01.237200+0000","last_peered":"2024-09-18T11:55:01.237200+0000","last_clean":"2024-09-18T11:55:01.237200+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:01.237200+0000","last_undegraded":"2024-09-18T11:55:01.237200+0000","last_fullsized":"2024-09-18T11:55:01.237200+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:14:25.540910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"34'3491","reported_seq":3554,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.297659+0000","last_change":"2024-09-18T11:54:58.212475+0000","last_active":"2024-09-18T11:55:01.297659+0000","last_peered":"2024-09-18T11:55:01.297659+0000","last_clean":"2024-09-18T11:55:01.297659+0000","last_became_active":"2024-09-18T11:54:58.207790+0000","last_became_peered":"2024-09-18T11:54:58.207790+0000","last_unstale":"2024-09-18T11:55:01.297659+0000","last_undegraded":"2024-09-18T11:55:01.297659+0000","last_fullsized":"2024-09-18T11:55:01.297659+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14267602,"num_objects":3485,"num_object_clones":0,"num_object_copies":6970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3485,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":13936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":75,"reported_epoch":33,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:54:58.213412+0000","last_change":"2024-09-18T11:54:58.212094+0000","last_active":"2024-09-18T11:54:58.213412+0000","last_peered":"2024-09-18T11:54:58.213412+0000","last_clean":"2024-09-18T11:54:58.213412+0000","last_became_active":"2024-09-18T11:54:58.207631+0000","last_became_peered":"2024-09-18T11:54:58.207631+0000","last_unstale":"2024-09-18T11:54:58.213412+0000","last_undegraded":"2024-09-18T11:54:58.213412+0000","last_fullsized":"2024-09-18T11:54:58.213412+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:53:31.760701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"34'3555","reported_seq":3599,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.241047+0000","last_change":"2024-09-18T11:55:02.241598+0000","last_active":"2024-09-18T11:55:02.241015+0000","last_peered":"2024-09-18T11:55:02.235429+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:54:58.219699+0000","last_became_peered":"2024-09-18T11:54:58.219699+0000","last_unstale":"2024-09-18T11:55:02.241047+0000","last_undegraded":"2024-09-18T11:55:02.241047+0000","last_fullsized":"2024-09-18T11:55:02.241047+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3555,"log_dups_size":0,"ondisk_log_size":3555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14542034,"num_objects":3552,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3553,"num_write_kb":14203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":39,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.251896+0000","last_change":"2024-09-18T11:55:02.252085+0000","last_active":"2024-09-18T11:55:02.251871+0000","last_peered":"2024-09-18T11:55:01.240094+0000","last_clean":"2024-09-18T11:55:01.240094+0000","last_became_active":"2024-09-18T11:54:58.214590+0000","last_became_peered":"2024-09-18T11:54:58.214590+0000","last_unstale":"2024-09-18T11:55:02.251896+0000","last_undegraded":"2024-09-18T11:55:02.251896+0000","last_fullsized":"2024-09-18T11:55:02.251896+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:48.512243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"34'3599","reported_seq":3796,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.243113+0000","last_change":"2024-09-18T11:55:02.247155+0000","last_active":"2024-09-18T11:55:02.243084+0000","last_peered":"2024-09-18T11:55:02.228397+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:54:58.203629+0000","last_became_peered":"2024-09-18T11:54:58.203629+0000","last_unstale":"2024-09-18T11:55:02.243113+0000","last_undegraded":"2024-09-18T11:55:02.243113+0000","last_fullsized":"2024-09-18T11:55:02.243113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:09:29.956333+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14716928,"num_objects":3593,"num_object_clones":0,"num_object_copies":7186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"34'3475","reported_seq":3543,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.309320+0000","last_change":"2024-09-18T11:54:58.268664+0000","last_active":"2024-09-18T11:55:01.309320+0000","last_peered":"2024-09-18T11:55:01.309320+0000","last_clean":"2024-09-18T11:55:01.309320+0000","last_became_active":"2024-09-18T11:54:58.244264+0000","last_became_peered":"2024-09-18T11:54:58.244264+0000","last_unstale":"2024-09-18T11:55:01.309320+0000","last_undegraded":"2024-09-18T11:55:01.309320+0000","last_fullsized":"2024-09-18T11:55:01.309320+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3475,"log_dups_size":0,"ondisk_log_size":3475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14226025,"num_objects":3474,"num_object_clones":0,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3474,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3474,"num_write_kb":13893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3474}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":33,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:54:58.221287+0000","last_change":"2024-09-18T11:54:58.214786+0000","last_active":"2024-09-18T11:54:58.221287+0000","last_peered":"2024-09-18T11:54:58.221287+0000","last_clean":"2024-09-18T11:54:58.221287+0000","last_became_active":"2024-09-18T11:54:58.214449+0000","last_became_peered":"2024-09-18T11:54:58.214449+0000","last_unstale":"2024-09-18T11:54:58.221287+0000","last_undegraded":"2024-09-18T11:54:58.221287+0000","last_fullsized":"2024-09-18T11:54:58.221287+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:22:21.910099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"34'3472","reported_seq":3542,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.311932+0000","last_change":"2024-09-18T11:54:58.216748+0000","last_active":"2024-09-18T11:55:01.311932+0000","last_peered":"2024-09-18T11:55:01.311932+0000","last_clean":"2024-09-18T11:55:01.311932+0000","last_became_active":"2024-09-18T11:54:58.209450+0000","last_became_peered":"2024-09-18T11:54:58.209450+0000","last_unstale":"2024-09-18T11:55:01.311932+0000","last_undegraded":"2024-09-18T11:55:01.311932+0000","last_fullsized":"2024-09-18T11:55:01.311932+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3472,"log_dups_size":0,"ondisk_log_size":3472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14190395,"num_objects":3467,"num_object_clones":0,"num_object_copies":6934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3467,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3469,"num_write_kb":13860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3467}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.242328+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:01.242328+0000","last_peered":"2024-09-18T11:55:01.242328+0000","last_clean":"2024-09-18T11:55:01.242328+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:01.242328+0000","last_undegraded":"2024-09-18T11:55:01.242328+0000","last_fullsized":"2024-09-18T11:55:01.242328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:10:36.497525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"34'3534","reported_seq":3601,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.321683+0000","last_change":"2024-09-18T11:54:58.213368+0000","last_active":"2024-09-18T11:55:01.321683+0000","last_peered":"2024-09-18T11:55:01.321683+0000","last_clean":"2024-09-18T11:55:01.321683+0000","last_became_active":"2024-09-18T11:54:58.208450+0000","last_became_peered":"2024-09-18T11:54:58.208450+0000","last_unstale":"2024-09-18T11:55:01.321683+0000","last_undegraded":"2024-09-18T11:55:01.321683+0000","last_fullsized":"2024-09-18T11:55:01.321683+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14460114,"num_objects":3532,"num_object_clones":0,"num_object_copies":7064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3532,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3532,"num_write_kb":14122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":33,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:54:58.223862+0000","last_change":"2024-09-18T11:54:58.215175+0000","last_active":"2024-09-18T11:54:58.223862+0000","last_peered":"2024-09-18T11:54:58.223862+0000","last_clean":"2024-09-18T11:54:58.223862+0000","last_became_active":"2024-09-18T11:54:58.207671+0000","last_became_peered":"2024-09-18T11:54:58.207671+0000","last_unstale":"2024-09-18T11:54:58.223862+0000","last_undegraded":"2024-09-18T11:54:58.223862+0000","last_fullsized":"2024-09-18T11:54:58.223862+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:45:00.297452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.246125+0000","last_change":"2024-09-18T11:55:02.246320+0000","last_active":"2024-09-18T11:55:02.244640+0000","last_peered":"2024-09-18T11:55:01.242430+0000","last_clean":"2024-09-18T11:55:01.242430+0000","last_became_active":"2024-09-18T11:54:58.229836+0000","last_became_peered":"2024-09-18T11:54:58.229836+0000","last_unstale":"2024-09-18T11:55:02.246125+0000","last_undegraded":"2024-09-18T11:55:02.246125+0000","last_fullsized":"2024-09-18T11:55:02.246125+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:28:51.494138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"34'3470","reported_seq":3541,"reported_epoch":34,"state":"active+clean+remapped","last_fresh":"2024-09-18T11:55:01.324185+0000","last_change":"2024-09-18T11:54:58.212484+0000","last_active":"2024-09-18T11:55:01.324185+0000","last_peered":"2024-09-18T11:55:01.324185+0000","last_clean":"2024-09-18T11:55:01.324185+0000","last_became_active":"2024-09-18T11:54:58.207705+0000","last_became_peered":"2024-09-18T11:54:58.207705+0000","last_unstale":"2024-09-18T11:55:01.324185+0000","last_undegraded":"2024-09-18T11:55:01.324185+0000","last_fullsized":"2024-09-18T11:55:01.324185+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3470,"log_dups_size":0,"ondisk_log_size":3470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14166436,"num_objects":3462,"num_object_clones":0,"num_object_copies":6924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3462,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3467,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":3462}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":230302365,"num_objects":56244,"num_object_clones":0,"num_object_copies":112488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27932,"num_objects_unfound":0,"num_objects_dirty":56244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43980,"num_write_kb":175797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":461791232,"data_stored":461686862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56305,"ondisk_log_size":56305,"up":24,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051598,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523532,"kb_used_data":225664,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93848308,"statfs":{"total":96636764160,"available":96100667392,"internally_reserved":0,"allocated":231079936,"data_stored":230877841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298180,"kb_used_data":316,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073660,"statfs":{"total":96636764160,"available":96331427840,"internally_reserved":0,"allocated":323584,"data_stored":77342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521608,"kb_used_data":223804,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93850232,"statfs":{"total":96636764160,"available":96102637568,"internally_reserved":0,"allocated":229175296,"data_stored":228973201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,9,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":228917248,"data_stored":228913769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":17276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":232759296,"data_stored":232755817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:07.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:07.101 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 41 20 58222 58202 5.54393 2.28516 0.0238811 0.014082 2024-09-18T11:55:07.677 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:07.703 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:07.718 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-18T11:55:07.039935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":235221063,"num_objects":57446,"num_object_clones":0,"num_object_copies":114892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41501,"num_write_kb":165878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57509,"ondisk_log_size":57509,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1355356,"kb_used_data":457656,"kb_used_omap":13,"kb_used_meta":897650,"kb_avail":281760164,"statfs":{"total":289910292480,"available":288522407936,"internally_reserved":0,"allocated":468639744,"data_stored":467927555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":919194130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,8,1,4,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20912325,"num_objects":5103,"num_object_clones":0,"num_object_copies":10206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2094,"num_write_kb":-8360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.812626"},"pg_stats":[{"pgid":"2.f","version":"37'3656","reported_seq":3724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.313372+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:06.313372+0000","last_peered":"2024-09-18T11:55:06.313372+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:06.313372+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3656,"log_dups_size":0,"ondisk_log_size":3656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14974976,"num_objects":3656,"num_object_clones":0,"num_object_copies":7312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3656,"num_write_kb":14624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":57},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"34'3558","reported_seq":3712,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:55:02.246157+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:02.246157+0000","last_peered":"2024-09-18T11:55:02.246157+0000","last_clean":"2024-09-18T11:55:02.246157+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:02.246157+0000","last_undegraded":"2024-09-18T11:55:02.246157+0000","last_fullsized":"2024-09-18T11:55:02.246157+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14548992,"num_objects":3552,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":750,"num_write_kb":3000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"34'3540","reported_seq":3713,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.246687+0000","last_change":"2024-09-18T11:55:02.249367+0000","last_active":"2024-09-18T11:55:02.246398+0000","last_peered":"2024-09-18T11:55:02.215905+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:54:58.199903+0000","last_became_peered":"2024-09-18T11:54:58.199903+0000","last_unstale":"2024-09-18T11:55:02.246687+0000","last_undegraded":"2024-09-18T11:55:02.246687+0000","last_fullsized":"2024-09-18T11:55:02.246687+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":3540,"log_dups_size":0,"ondisk_log_size":3540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:26:33.136656+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14483456,"num_objects":3536,"num_object_clones":0,"num_object_copies":7072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":398,"num_write_kb":1592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"34'3503","reported_seq":3528,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T11:55:02.240529+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:02.240529+0000","last_peered":"2024-09-18T11:55:02.240529+0000","last_clean":"2024-09-18T11:55:02.240529+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:02.240529+0000","last_undegraded":"2024-09-18T11:55:02.240529+0000","last_fullsized":"2024-09-18T11:55:02.240529+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14348288,"num_objects":3503,"num_object_clones":0,"num_object_copies":7006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":14012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'3599","reported_seq":3788,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:06.330046+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:06.330046+0000","last_peered":"2024-09-18T11:55:06.330046+0000","last_clean":"2024-09-18T11:55:06.330046+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:06.330046+0000","last_undegraded":"2024-09-18T11:55:06.330046+0000","last_fullsized":"2024-09-18T11:55:06.330046+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14712832,"num_objects":3592,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":764,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'3584","reported_seq":3663,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.312544+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:06.312544+0000","last_peered":"2024-09-18T11:55:06.312544+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:06.312544+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14660818,"num_objects":3581,"num_object_clones":0,"num_object_copies":7162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3582,"num_write_kb":14319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":54},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'3633","reported_seq":3718,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.335210+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:06.335210+0000","last_peered":"2024-09-18T11:55:06.335210+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:06.335210+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857426,"num_objects":3629,"num_object_clones":0,"num_object_copies":7258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3631,"num_write_kb":14512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":70},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'3736","reported_seq":3959,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:06.343839+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:06.343839+0000","last_peered":"2024-09-18T11:55:06.343839+0000","last_clean":"2024-09-18T11:55:06.343839+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:06.343839+0000","last_undegraded":"2024-09-18T11:55:06.343839+0000","last_fullsized":"2024-09-18T11:55:06.343839+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15294464,"num_objects":3734,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53,"num_write_kb":212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"34'3486","reported_seq":3627,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.247293+0000","last_change":"2024-09-18T11:55:02.249595+0000","last_active":"2024-09-18T11:55:02.247234+0000","last_peered":"2024-09-18T11:55:02.234203+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:54:58.199897+0000","last_became_peered":"2024-09-18T11:54:58.199897+0000","last_unstale":"2024-09-18T11:55:02.247293+0000","last_undegraded":"2024-09-18T11:55:02.247293+0000","last_fullsized":"2024-09-18T11:55:02.247293+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":3486,"log_dups_size":0,"ondisk_log_size":3486,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14251218,"num_objects":3481,"num_object_clones":0,"num_object_copies":6962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":13920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.237200+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:01.237200+0000","last_peered":"2024-09-18T11:55:01.237200+0000","last_clean":"2024-09-18T11:55:01.237200+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:01.237200+0000","last_undegraded":"2024-09-18T11:55:01.237200+0000","last_fullsized":"2024-09-18T11:55:01.237200+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:14:25.540910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'3602","reported_seq":3678,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.298554+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:06.298554+0000","last_peered":"2024-09-18T11:55:06.298554+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:06.298554+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3602,"log_dups_size":0,"ondisk_log_size":3602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14722258,"num_objects":3596,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3600,"num_write_kb":14380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":64},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"34'3555","reported_seq":3599,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.241047+0000","last_change":"2024-09-18T11:55:02.241598+0000","last_active":"2024-09-18T11:55:02.241015+0000","last_peered":"2024-09-18T11:55:02.235429+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:54:58.219699+0000","last_became_peered":"2024-09-18T11:54:58.219699+0000","last_unstale":"2024-09-18T11:55:02.241047+0000","last_undegraded":"2024-09-18T11:55:02.241047+0000","last_fullsized":"2024-09-18T11:55:02.241047+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3555,"log_dups_size":0,"ondisk_log_size":3555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14542034,"num_objects":3552,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3553,"num_write_kb":14203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"34'3599","reported_seq":3796,"reported_epoch":35,"state":"remapped+peering","last_fresh":"2024-09-18T11:55:02.243113+0000","last_change":"2024-09-18T11:55:02.247155+0000","last_active":"2024-09-18T11:55:02.243084+0000","last_peered":"2024-09-18T11:55:02.228397+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:54:58.203629+0000","last_became_peered":"2024-09-18T11:54:58.203629+0000","last_unstale":"2024-09-18T11:55:02.243113+0000","last_undegraded":"2024-09-18T11:55:02.243113+0000","last_fullsized":"2024-09-18T11:55:02.243113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:09:29.956333+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14716928,"num_objects":3593,"num_object_clones":0,"num_object_copies":7186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'3599","reported_seq":3679,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.309431+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:06.309431+0000","last_peered":"2024-09-18T11:55:06.309431+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:06.309431+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14733929,"num_objects":3598,"num_object_clones":0,"num_object_copies":7196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3598,"num_write_kb":14389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":63},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'3613","reported_seq":3690,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.294939+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:06.294939+0000","last_peered":"2024-09-18T11:55:06.294939+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:06.294939+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14767931,"num_objects":3608,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":14424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":86},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T11:55:01.242328+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:01.242328+0000","last_peered":"2024-09-18T11:55:01.242328+0000","last_clean":"2024-09-18T11:55:01.242328+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:01.242328+0000","last_undegraded":"2024-09-18T11:55:01.242328+0000","last_fullsized":"2024-09-18T11:55:01.242328+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:10:36.497525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'3653","reported_seq":3732,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.312503+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:06.312503+0000","last_peered":"2024-09-18T11:55:06.312503+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:06.312503+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3653,"log_dups_size":0,"ondisk_log_size":3653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14947538,"num_objects":3651,"num_object_clones":0,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3651,"num_write_kb":14598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":58},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'3590","reported_seq":3673,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.283015+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:06.283015+0000","last_peered":"2024-09-18T11:55:06.283015+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:06.283015+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3590,"log_dups_size":0,"ondisk_log_size":3590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14657956,"num_objects":3582,"num_object_clones":0,"num_object_copies":7164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":14319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":59},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":235221044,"num_objects":57444,"num_object_clones":0,"num_object_copies":114888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41499,"num_write_kb":165876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":8809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471527424,"data_stored":471440449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57506,"ondisk_log_size":57506,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051598,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523532,"kb_used_data":225664,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93848308,"statfs":{"total":96636764160,"available":96100667392,"internally_reserved":0,"allocated":231079936,"data_stored":230877841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855362,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299404,"kb_used_data":1536,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94072436,"statfs":{"total":96636764160,"available":96330174464,"internally_reserved":0,"allocated":1572864,"data_stored":1279548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532420,"kb_used_data":230456,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93839420,"statfs":{"total":96636764160,"available":96091566080,"internally_reserved":0,"allocated":235986944,"data_stored":235770166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,8,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":235724800,"data_stored":235721321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3043328,"data_stored":2963311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":232759296,"data_stored":232755817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:07.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:55:07.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'3656", 'reported_seq': 3724, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:06.313372+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:55:06.313372+0000', 'last_peered': '2024-09-18T11:55:06.313372+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:55:06.313372+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3656, 'log_dups_size': 0, 'ondisk_log_size': 3656, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14974976, 'num_objects': 3656, 'num_object_clones': 0, 'num_object_copies': 7312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3656, 'num_write_kb': 14624, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 57}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:07.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:55:07.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "34'3540", 'reported_seq': 3713, 'reported_epoch': 35, 'state': 'remapped+peering', 'last_fresh': '2024-09-18T11:55:02.246687+0000', 'last_change': '2024-09-18T11:55:02.249367+0000', 'last_active': '2024-09-18T11:55:02.246398+0000', 'last_peered': '2024-09-18T11:55:02.215905+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:54:58.199903+0000', 'last_became_peered': '2024-09-18T11:54:58.199903+0000', 'last_unstale': '2024-09-18T11:55:02.246687+0000', 'last_undegraded': '2024-09-18T11:55:02.246687+0000', 'last_fullsized': '2024-09-18T11:55:02.246687+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 3540, 'log_dups_size': 0, 'ondisk_log_size': 3540, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:26:33.136656+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14483456, 'num_objects': 3536, 'num_object_clones': 0, 'num_object_copies': 7072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 398, 'num_write_kb': 1592, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:07.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:55:07.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'3584", 'reported_seq': 3663, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:06.312544+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:55:06.312544+0000', 'last_peered': '2024-09-18T11:55:06.312544+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:55:06.312544+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3584, 'log_dups_size': 0, 'ondisk_log_size': 3584, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14660818, 'num_objects': 3581, 'num_object_clones': 0, 'num_object_copies': 7162, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3582, 'num_write_kb': 14319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 54}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:07.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:55:07.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'3633", 'reported_seq': 3718, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:06.335210+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:55:06.335210+0000', 'last_peered': '2024-09-18T11:55:06.335210+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:55:06.335210+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3633, 'log_dups_size': 0, 'ondisk_log_size': 3633, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14857426, 'num_objects': 3629, 'num_object_clones': 0, 'num_object_copies': 7258, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3631, 'num_write_kb': 14512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 70}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:07.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:55:07.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "34'3486", 'reported_seq': 3627, 'reported_epoch': 35, 'state': 'remapped+peering', 'last_fresh': '2024-09-18T11:55:02.247293+0000', 'last_change': '2024-09-18T11:55:02.249595+0000', 'last_active': '2024-09-18T11:55:02.247234+0000', 'last_peered': '2024-09-18T11:55:02.234203+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:54:58.199897+0000', 'last_became_peered': '2024-09-18T11:54:58.199897+0000', 'last_unstale': '2024-09-18T11:55:02.247293+0000', 'last_undegraded': '2024-09-18T11:55:02.247293+0000', 'last_fullsized': '2024-09-18T11:55:02.247293+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 3486, 'log_dups_size': 0, 'ondisk_log_size': 3486, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14251218, 'num_objects': 3481, 'num_object_clones': 0, 'num_object_copies': 6962, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3484, 'num_write_kb': 13920, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:07.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:55:07.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'3602", 'reported_seq': 3678, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:06.298554+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:55:06.298554+0000', 'last_peered': '2024-09-18T11:55:06.298554+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:55:06.298554+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3602, 'log_dups_size': 0, 'ondisk_log_size': 3602, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14722258, 'num_objects': 3596, 'num_object_clones': 0, 'num_object_copies': 7192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3600, 'num_write_kb': 14380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 64}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:07.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:55:07.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "34'3555", 'reported_seq': 3599, 'reported_epoch': 35, 'state': 'remapped+peering', 'last_fresh': '2024-09-18T11:55:02.241047+0000', 'last_change': '2024-09-18T11:55:02.241598+0000', 'last_active': '2024-09-18T11:55:02.241015+0000', 'last_peered': '2024-09-18T11:55:02.235429+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:54:58.219699+0000', 'last_became_peered': '2024-09-18T11:54:58.219699+0000', 'last_unstale': '2024-09-18T11:55:02.241047+0000', 'last_undegraded': '2024-09-18T11:55:02.241047+0000', 'last_fullsized': '2024-09-18T11:55:02.241047+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3555, 'log_dups_size': 0, 'ondisk_log_size': 3555, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14542034, 'num_objects': 3552, 'num_object_clones': 0, 'num_object_copies': 7104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3553, 'num_write_kb': 14203, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:07.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:55:07.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "34'3599", 'reported_seq': 3796, 'reported_epoch': 35, 'state': 'remapped+peering', 'last_fresh': '2024-09-18T11:55:02.243113+0000', 'last_change': '2024-09-18T11:55:02.247155+0000', 'last_active': '2024-09-18T11:55:02.243084+0000', 'last_peered': '2024-09-18T11:55:02.228397+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:54:58.203629+0000', 'last_became_peered': '2024-09-18T11:54:58.203629+0000', 'last_unstale': '2024-09-18T11:55:02.243113+0000', 'last_undegraded': '2024-09-18T11:55:02.243113+0000', 'last_fullsized': '2024-09-18T11:55:02.243113+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 3599, 'log_dups_size': 0, 'ondisk_log_size': 3599, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:09:29.956333+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14716928, 'num_objects': 3593, 'num_object_clones': 0, 'num_object_copies': 7186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 79, 'num_write_kb': 316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:07.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:55:07.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'3599", 'reported_seq': 3679, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:06.309431+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:55:06.309431+0000', 'last_peered': '2024-09-18T11:55:06.309431+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:55:06.309431+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3599, 'log_dups_size': 0, 'ondisk_log_size': 3599, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14733929, 'num_objects': 3598, 'num_object_clones': 0, 'num_object_copies': 7196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3598, 'num_write_kb': 14389, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 63}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:07.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:55:07.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'3613", 'reported_seq': 3690, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:06.294939+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:55:06.294939+0000', 'last_peered': '2024-09-18T11:55:06.294939+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:55:06.294939+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3613, 'log_dups_size': 0, 'ondisk_log_size': 3613, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14767931, 'num_objects': 3608, 'num_object_clones': 0, 'num_object_copies': 7216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3610, 'num_write_kb': 14424, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 86}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:07.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:55:07.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'3653", 'reported_seq': 3732, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:06.312503+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:55:06.312503+0000', 'last_peered': '2024-09-18T11:55:06.312503+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:55:06.312503+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3653, 'log_dups_size': 0, 'ondisk_log_size': 3653, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14947538, 'num_objects': 3651, 'num_object_clones': 0, 'num_object_copies': 7302, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3651, 'num_write_kb': 14598, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 58}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:07.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:55:07.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'3590", 'reported_seq': 3673, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:06.283015+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:55:06.283015+0000', 'last_peered': '2024-09-18T11:55:06.283015+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:55:06.283015+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3590, 'log_dups_size': 0, 'ondisk_log_size': 3590, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14657956, 'num_objects': 3582, 'num_object_clones': 0, 'num_object_copies': 7164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3587, 'num_write_kb': 14319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 59}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:07.722 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:08.101 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 42 20 58807 58787 5.46634 2.28516 0.00987257 0.0142807 2024-09-18T11:55:08.514 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:08.514 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28405/114892 objects degraded (24.723%), 8 pgs degraded","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":108,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"remapped+peering","count":4}],"num_pgs":24,"num_pools":2,"num_objects":57446,"data_bytes":235221063,"bytes_used":1387884544,"bytes_avail":288522407936,"bytes_total":289910292480,"inactive_pgs_ratio":0.1666666716337204,"degraded_objects":28405,"degraded_total":114892,"degraded_ratio":0.2472321832677645,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":1409,"recovering_keys_per_sec":0,"num_objects_recovered":0,"num_bytes_recovered":8192,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T11:55:03.038775+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:55:08.550 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:08.550 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:09.101 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 43 20 59400 59380 5.39309 2.31641 0.0290681 0.0144751 2024-09-18T11:55:09.199 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:09.215 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:09.225 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-18T11:55:09.040329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":237637703,"num_objects":58036,"num_object_clones":0,"num_object_copies":116072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3420,"num_objects_unfound":0,"num_objects_dirty":58036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42091,"num_write_kb":168238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":246396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58099,"ondisk_log_size":58099,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1363796,"kb_used_data":461936,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281751724,"statfs":{"total":289910292480,"available":288513765376,"internally_reserved":0,"allocated":473022464,"data_stored":472299214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,8,1,4,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14055621,"num_objects":3429,"num_object_clones":0,"num_object_copies":6858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":-24513,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3768,"num_write_kb":-15056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":237568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.811551"},"pg_stats":[{"pgid":"2.f","version":"37'3656","reported_seq":3724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.313372+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:06.313372+0000","last_peered":"2024-09-18T11:55:06.313372+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:06.313372+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3656,"log_dups_size":0,"ondisk_log_size":3656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14974976,"num_objects":3656,"num_object_clones":0,"num_object_copies":7312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3656,"num_write_kb":14624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":57},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'3652","reported_seq":3808,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:07.377160+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:07.377160+0000","last_peered":"2024-09-18T11:55:07.377160+0000","last_clean":"2024-09-18T11:55:07.377160+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:07.377160+0000","last_undegraded":"2024-09-18T11:55:07.377160+0000","last_fullsized":"2024-09-18T11:55:07.377160+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":3652,"log_dups_size":0,"ondisk_log_size":3652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14934016,"num_objects":3646,"num_object_clones":0,"num_object_copies":7292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":844,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'3642","reported_seq":3825,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:07.414258+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:07.414258+0000","last_peered":"2024-09-18T11:55:07.414258+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:07.414258+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":3642,"log_dups_size":0,"ondisk_log_size":3642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14901248,"num_objects":3638,"num_object_clones":0,"num_object_copies":7276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":500,"num_write_kb":2000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'3603","reported_seq":3630,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:07.397793+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:07.397793+0000","last_peered":"2024-09-18T11:55:07.397793+0000","last_clean":"2024-09-18T11:55:07.397793+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:07.397793+0000","last_undegraded":"2024-09-18T11:55:07.397793+0000","last_fullsized":"2024-09-18T11:55:07.397793+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3603,"log_dups_size":0,"ondisk_log_size":3603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14757888,"num_objects":3603,"num_object_clones":0,"num_object_copies":7206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3603,"num_write_kb":14412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'3599","reported_seq":3788,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:06.330046+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:06.330046+0000","last_peered":"2024-09-18T11:55:06.330046+0000","last_clean":"2024-09-18T11:55:06.330046+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:06.330046+0000","last_undegraded":"2024-09-18T11:55:06.330046+0000","last_fullsized":"2024-09-18T11:55:06.330046+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14712832,"num_objects":3592,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":764,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'3584","reported_seq":3663,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.312544+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:06.312544+0000","last_peered":"2024-09-18T11:55:06.312544+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:06.312544+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14660818,"num_objects":3581,"num_object_clones":0,"num_object_copies":7162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3582,"num_write_kb":14319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":54},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'3633","reported_seq":3718,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.335210+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:06.335210+0000","last_peered":"2024-09-18T11:55:06.335210+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:06.335210+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857426,"num_objects":3629,"num_object_clones":0,"num_object_copies":7258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3631,"num_write_kb":14512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":70},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'3736","reported_seq":3959,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:06.343839+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:06.343839+0000","last_peered":"2024-09-18T11:55:06.343839+0000","last_clean":"2024-09-18T11:55:06.343839+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:06.343839+0000","last_undegraded":"2024-09-18T11:55:06.343839+0000","last_fullsized":"2024-09-18T11:55:06.343839+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15294464,"num_objects":3734,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53,"num_write_kb":212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'3590","reported_seq":3856,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:07.454247+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:07.454247+0000","last_peered":"2024-09-18T11:55:07.454247+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:07.454247+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":3590,"log_dups_size":0,"ondisk_log_size":3590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14677202,"num_objects":3585,"num_object_clones":0,"num_object_copies":7170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3420,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3588,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":237568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3420},{"shards":"2","objects":164}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'3602","reported_seq":3678,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.298554+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:06.298554+0000","last_peered":"2024-09-18T11:55:06.298554+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:06.298554+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3602,"log_dups_size":0,"ondisk_log_size":3602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14722258,"num_objects":3596,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3600,"num_write_kb":14380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":64},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'3657","reported_seq":3708,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:07.432072+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:07.432072+0000","last_peered":"2024-09-18T11:55:07.432072+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:07.432072+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3657,"log_dups_size":0,"ondisk_log_size":3657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14959826,"num_objects":3654,"num_object_clones":0,"num_object_copies":7308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3655,"num_write_kb":14611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'3687","reported_seq":3894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:07.431859+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:07.431859+0000","last_peered":"2024-09-18T11:55:07.431859+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:07.431859+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15077376,"num_objects":3681,"num_object_clones":0,"num_object_copies":7362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":167,"num_write_kb":668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":88}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'3599","reported_seq":3679,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.309431+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:06.309431+0000","last_peered":"2024-09-18T11:55:06.309431+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:06.309431+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14733929,"num_objects":3598,"num_object_clones":0,"num_object_copies":7196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3598,"num_write_kb":14389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":63},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'3613","reported_seq":3690,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.294939+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:06.294939+0000","last_peered":"2024-09-18T11:55:06.294939+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:06.294939+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14767931,"num_objects":3608,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":14424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":86},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'3653","reported_seq":3732,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.312503+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:06.312503+0000","last_peered":"2024-09-18T11:55:06.312503+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:06.312503+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3653,"log_dups_size":0,"ondisk_log_size":3653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14947538,"num_objects":3651,"num_object_clones":0,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3651,"num_write_kb":14598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":58},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'3590","reported_seq":3673,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.283015+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:06.283015+0000","last_peered":"2024-09-18T11:55:06.283015+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:06.283015+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3590,"log_dups_size":0,"ondisk_log_size":3590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14657956,"num_objects":3582,"num_object_clones":0,"num_object_copies":7164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":14319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":59},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":237637684,"num_objects":58034,"num_object_clones":0,"num_object_copies":116068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3420,"num_objects_unfound":0,"num_objects_dirty":58034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42089,"num_write_kb":168236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":246377,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":474714112,"data_stored":474627137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58096,"ondisk_log_size":58096,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051599,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531972,"kb_used_data":229944,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93839868,"statfs":{"total":96636764160,"available":96092024832,"internally_reserved":0,"allocated":235462656,"data_stored":235249500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855362,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299404,"kb_used_data":1536,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94072436,"statfs":{"total":96636764160,"available":96330174464,"internally_reserved":0,"allocated":1572864,"data_stored":1279548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532420,"kb_used_data":230456,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93839420,"statfs":{"total":96636764160,"available":96091566080,"internally_reserved":0,"allocated":235986944,"data_stored":235770166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,8,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":235724800,"data_stored":235721321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3043328,"data_stored":2963311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":235945984,"data_stored":235942505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:10.103 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 44 20 60011 59991 5.32457 2.38672 0.0672931 0.0146618 2024-09-18T11:55:11.103 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 45 20 60666 60646 5.26309 2.55859 0.0288844 0.014835 2024-09-18T11:55:12.103 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 46 20 61303 61283 5.20277 2.48828 0.0550056 0.0150058 2024-09-18T11:55:12.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:12.830 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:12.835 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:12.872 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-18T11:55:11.040859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":237637703,"num_objects":58036,"num_object_clones":0,"num_object_copies":116072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3420,"num_objects_unfound":0,"num_objects_dirty":58036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42091,"num_write_kb":168238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":246396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58099,"ondisk_log_size":58099,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1363796,"kb_used_data":461936,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281751724,"statfs":{"total":289910292480,"available":288513765376,"internally_reserved":0,"allocated":473022464,"data_stored":472299214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,8,1,4,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7335319,"num_objects":1790,"num_object_clones":0,"num_object_copies":3580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":-24513,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1891,"num_write_kb":-7561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":237568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002203"},"pg_stats":[{"pgid":"2.f","version":"37'3656","reported_seq":3724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.313372+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:06.313372+0000","last_peered":"2024-09-18T11:55:06.313372+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:06.313372+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3656,"log_dups_size":0,"ondisk_log_size":3656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14974976,"num_objects":3656,"num_object_clones":0,"num_object_copies":7312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3656,"num_write_kb":14624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":57},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'3652","reported_seq":3808,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:07.377160+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:07.377160+0000","last_peered":"2024-09-18T11:55:07.377160+0000","last_clean":"2024-09-18T11:55:07.377160+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:07.377160+0000","last_undegraded":"2024-09-18T11:55:07.377160+0000","last_fullsized":"2024-09-18T11:55:07.377160+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":3652,"log_dups_size":0,"ondisk_log_size":3652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14934016,"num_objects":3646,"num_object_clones":0,"num_object_copies":7292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":844,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'3642","reported_seq":3825,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:07.414258+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:07.414258+0000","last_peered":"2024-09-18T11:55:07.414258+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:07.414258+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":3642,"log_dups_size":0,"ondisk_log_size":3642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14901248,"num_objects":3638,"num_object_clones":0,"num_object_copies":7276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":500,"num_write_kb":2000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":102}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'3603","reported_seq":3630,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:07.397793+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:07.397793+0000","last_peered":"2024-09-18T11:55:07.397793+0000","last_clean":"2024-09-18T11:55:07.397793+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:07.397793+0000","last_undegraded":"2024-09-18T11:55:07.397793+0000","last_fullsized":"2024-09-18T11:55:07.397793+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3603,"log_dups_size":0,"ondisk_log_size":3603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14757888,"num_objects":3603,"num_object_clones":0,"num_object_copies":7206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3603,"num_write_kb":14412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'3599","reported_seq":3788,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:06.330046+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:06.330046+0000","last_peered":"2024-09-18T11:55:06.330046+0000","last_clean":"2024-09-18T11:55:06.330046+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:06.330046+0000","last_undegraded":"2024-09-18T11:55:06.330046+0000","last_fullsized":"2024-09-18T11:55:06.330046+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14712832,"num_objects":3592,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":764,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'3584","reported_seq":3663,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.312544+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:06.312544+0000","last_peered":"2024-09-18T11:55:06.312544+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:06.312544+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14660818,"num_objects":3581,"num_object_clones":0,"num_object_copies":7162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3582,"num_write_kb":14319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":54},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'3633","reported_seq":3718,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.335210+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:06.335210+0000","last_peered":"2024-09-18T11:55:06.335210+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:06.335210+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14857426,"num_objects":3629,"num_object_clones":0,"num_object_copies":7258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3631,"num_write_kb":14512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":70},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'3736","reported_seq":3959,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:06.343839+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:06.343839+0000","last_peered":"2024-09-18T11:55:06.343839+0000","last_clean":"2024-09-18T11:55:06.343839+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:06.343839+0000","last_undegraded":"2024-09-18T11:55:06.343839+0000","last_fullsized":"2024-09-18T11:55:06.343839+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15294464,"num_objects":3734,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53,"num_write_kb":212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'3590","reported_seq":3856,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:07.454247+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:07.454247+0000","last_peered":"2024-09-18T11:55:07.454247+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:07.454247+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":3590,"log_dups_size":0,"ondisk_log_size":3590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14677202,"num_objects":3585,"num_object_clones":0,"num_object_copies":7170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3420,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3588,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":237568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3420},{"shards":"2","objects":164}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'3602","reported_seq":3678,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.298554+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:06.298554+0000","last_peered":"2024-09-18T11:55:06.298554+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:06.298554+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3602,"log_dups_size":0,"ondisk_log_size":3602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14722258,"num_objects":3596,"num_object_clones":0,"num_object_copies":7192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3600,"num_write_kb":14380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":64},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'3657","reported_seq":3708,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:07.432072+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:07.432072+0000","last_peered":"2024-09-18T11:55:07.432072+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:07.432072+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3657,"log_dups_size":0,"ondisk_log_size":3657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14959826,"num_objects":3654,"num_object_clones":0,"num_object_copies":7308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3655,"num_write_kb":14611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'3687","reported_seq":3894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:07.431859+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:07.431859+0000","last_peered":"2024-09-18T11:55:07.431859+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:07.431859+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15077376,"num_objects":3681,"num_object_clones":0,"num_object_copies":7362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":167,"num_write_kb":668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":88}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'3599","reported_seq":3679,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.309431+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:06.309431+0000","last_peered":"2024-09-18T11:55:06.309431+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:06.309431+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14733929,"num_objects":3598,"num_object_clones":0,"num_object_copies":7196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3598,"num_write_kb":14389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":63},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'3613","reported_seq":3690,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.294939+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:06.294939+0000","last_peered":"2024-09-18T11:55:06.294939+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:06.294939+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14767931,"num_objects":3608,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":14424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":86},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'3653","reported_seq":3732,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.312503+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:06.312503+0000","last_peered":"2024-09-18T11:55:06.312503+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:06.312503+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3653,"log_dups_size":0,"ondisk_log_size":3653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14947538,"num_objects":3651,"num_object_clones":0,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3651,"num_write_kb":14598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":58},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'3590","reported_seq":3673,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:06.283015+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:06.283015+0000","last_peered":"2024-09-18T11:55:06.283015+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:06.283015+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3590,"log_dups_size":0,"ondisk_log_size":3590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14657956,"num_objects":3582,"num_object_clones":0,"num_object_copies":7164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":14319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":59},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":237637684,"num_objects":58034,"num_object_clones":0,"num_object_copies":116068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3420,"num_objects_unfound":0,"num_objects_dirty":58034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42089,"num_write_kb":168236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":246377,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":474714112,"data_stored":474627137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58096,"ondisk_log_size":58096,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051599,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531972,"kb_used_data":229944,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93839868,"statfs":{"total":96636764160,"available":96092024832,"internally_reserved":0,"allocated":235462656,"data_stored":235249500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855362,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299404,"kb_used_data":1536,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94072436,"statfs":{"total":96636764160,"available":96330174464,"internally_reserved":0,"allocated":1572864,"data_stored":1279548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532420,"kb_used_data":230456,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93839420,"statfs":{"total":96636764160,"available":96091566080,"internally_reserved":0,"allocated":235986944,"data_stored":235770166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,8,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.249}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":235724800,"data_stored":235721321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3043328,"data_stored":2963311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":235945984,"data_stored":235942505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:12.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:13.103 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 47 20 61897 61877 5.14145 2.32031 0.0371407 0.0151835 2024-09-18T11:55:13.504 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:13.519 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:13.538 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-18T11:55:13.041253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":250261575,"num_objects":61118,"num_object_clones":0,"num_object_copies":122236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3294,"num_objects_unfound":0,"num_objects_dirty":61118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45173,"num_write_kb":180566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":762492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61181,"ondisk_log_size":61181,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1405076,"kb_used_data":486588,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281710444,"statfs":{"total":289910292480,"available":288471494656,"internally_reserved":0,"allocated":498266112,"data_stored":497542862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,6,9,5,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19959191,"num_objects":4872,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":-24639,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1191,"num_write_kb":4767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.801986"},"pg_stats":[{"pgid":"2.f","version":"37'3845","reported_seq":3914,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.264390+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:11.264390+0000","last_peered":"2024-09-18T11:55:11.264390+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:11.264390+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15749120,"num_objects":3845,"num_object_clones":0,"num_object_copies":7690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3845,"num_write_kb":15380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":246},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'3858","reported_seq":4014,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:12.437938+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:12.437938+0000","last_peered":"2024-09-18T11:55:12.437938+0000","last_clean":"2024-09-18T11:55:12.437938+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:12.437938+0000","last_undegraded":"2024-09-18T11:55:12.437938+0000","last_fullsized":"2024-09-18T11:55:12.437938+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":3858,"log_dups_size":0,"ondisk_log_size":3858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15777792,"num_objects":3852,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1050,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'3829","reported_seq":4012,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:12.450820+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:12.450820+0000","last_peered":"2024-09-18T11:55:12.450820+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:12.450820+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15667200,"num_objects":3825,"num_object_clones":0,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":687,"num_write_kb":2748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":289}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'3780","reported_seq":3806,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:12.468320+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:12.468320+0000","last_peered":"2024-09-18T11:55:12.468320+0000","last_clean":"2024-09-18T11:55:12.468320+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:12.468320+0000","last_undegraded":"2024-09-18T11:55:12.468320+0000","last_fullsized":"2024-09-18T11:55:12.468320+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3780,"log_dups_size":0,"ondisk_log_size":3780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15482880,"num_objects":3780,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":15120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'3778","reported_seq":3967,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:11.307364+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:11.307364+0000","last_peered":"2024-09-18T11:55:11.307364+0000","last_clean":"2024-09-18T11:55:11.307364+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:11.307364+0000","last_undegraded":"2024-09-18T11:55:11.307364+0000","last_fullsized":"2024-09-18T11:55:11.307364+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15446016,"num_objects":3771,"num_object_clones":0,"num_object_copies":7542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":943,"num_write_kb":3772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'3786","reported_seq":3862,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.340902+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:11.340902+0000","last_peered":"2024-09-18T11:55:11.340902+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:11.340902+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15488210,"num_objects":3783,"num_object_clones":0,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":15127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":256},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'3815","reported_seq":3900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.298825+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:11.298825+0000","last_peered":"2024-09-18T11:55:11.298825+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:11.298825+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15602898,"num_objects":3811,"num_object_clones":0,"num_object_copies":7622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3813,"num_write_kb":15240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":252},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'3933","reported_seq":4156,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:11.354071+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:11.354071+0000","last_peered":"2024-09-18T11:55:11.354071+0000","last_clean":"2024-09-18T11:55:11.354071+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:11.354071+0000","last_undegraded":"2024-09-18T11:55:11.354071+0000","last_fullsized":"2024-09-18T11:55:11.354071+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16101376,"num_objects":3931,"num_object_clones":0,"num_object_copies":7862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":250,"num_write_kb":1000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'3765","reported_seq":4285,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:12.459137+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:12.459137+0000","last_peered":"2024-09-18T11:55:12.459137+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:12.459137+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":3765,"log_dups_size":0,"ondisk_log_size":3765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15394002,"num_objects":3760,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3294,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3763,"num_write_kb":15036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3294},{"shards":"2","objects":465}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'3782","reported_seq":3856,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.340840+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:11.340840+0000","last_peered":"2024-09-18T11:55:11.340840+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:11.340840+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15459538,"num_objects":3776,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":15100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":244},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'3850","reported_seq":3904,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:12.461468+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:12.461468+0000","last_peered":"2024-09-18T11:55:12.461468+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:12.461468+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15750354,"num_objects":3847,"num_object_clones":0,"num_object_copies":7694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":15383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'3895","reported_seq":4102,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:12.402347+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:12.402347+0000","last_peered":"2024-09-18T11:55:12.402347+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:12.402347+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":3895,"log_dups_size":0,"ondisk_log_size":3895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15929344,"num_objects":3889,"num_object_clones":0,"num_object_copies":7778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":375,"num_write_kb":1500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":296}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'3800","reported_seq":3880,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.217995+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:11.217995+0000","last_peered":"2024-09-18T11:55:11.217995+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:11.217995+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3800,"log_dups_size":0,"ondisk_log_size":3800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15557225,"num_objects":3799,"num_object_clones":0,"num_object_copies":7598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3799,"num_write_kb":15193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":264},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'3815","reported_seq":3897,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.325439+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:11.325439+0000","last_peered":"2024-09-18T11:55:11.325439+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:11.325439+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15595323,"num_objects":3810,"num_object_clones":0,"num_object_copies":7620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3812,"num_write_kb":15232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":288},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'3855","reported_seq":3934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.300697+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:11.300697+0000","last_peered":"2024-09-18T11:55:11.300697+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:11.300697+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15774930,"num_objects":3853,"num_object_clones":0,"num_object_copies":7706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3853,"num_write_kb":15406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":260},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'3792","reported_seq":3875,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.289392+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:11.289392+0000","last_peered":"2024-09-18T11:55:11.289392+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:11.289392+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3792,"log_dups_size":0,"ondisk_log_size":3792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15485348,"num_objects":3784,"num_object_clones":0,"num_object_copies":7568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":15127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":261},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":250261556,"num_objects":61116,"num_object_clones":0,"num_object_copies":122232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3294,"num_objects_unfound":0,"num_objects_dirty":61116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45171,"num_write_kb":180564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":762473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":500342784,"data_stored":500255809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61178,"ondisk_log_size":61178,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051600,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542008,"kb_used_data":235824,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93829832,"statfs":{"total":96636764160,"available":96081747968,"internally_reserved":0,"allocated":241483776,"data_stored":241270620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.183}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]}]},{"osd":1,"up_from":35,"seq":150323855363,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319212,"kb_used_data":13028,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94052628,"statfs":{"total":96636764160,"available":96309891072,"internally_reserved":0,"allocated":13340672,"data_stored":13047356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6680000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543856,"kb_used_data":237736,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93827984,"statfs":{"total":96636764160,"available":96079855616,"internally_reserved":0,"allocated":243441664,"data_stored":243224886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,5,3,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245186560,"data_stored":245183081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13025280,"data_stored":12945263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242130944,"data_stored":242127465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:13.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:55:13.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'3845", 'reported_seq': 3914, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:11.264390+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:55:11.264390+0000', 'last_peered': '2024-09-18T11:55:11.264390+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:55:11.264390+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3845, 'log_dups_size': 0, 'ondisk_log_size': 3845, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15749120, 'num_objects': 3845, 'num_object_clones': 0, 'num_object_copies': 7690, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3845, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3845, 'num_write_kb': 15380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 246}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:13.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:55:13.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'3829", 'reported_seq': 4012, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:12.450820+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:55:12.450820+0000', 'last_peered': '2024-09-18T11:55:12.450820+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:55:12.450820+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 3829, 'log_dups_size': 0, 'ondisk_log_size': 3829, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15667200, 'num_objects': 3825, 'num_object_clones': 0, 'num_object_copies': 7650, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 687, 'num_write_kb': 2748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 289}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:13.540 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:55:13.540 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'3786", 'reported_seq': 3862, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:11.340902+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:55:11.340902+0000', 'last_peered': '2024-09-18T11:55:11.340902+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:55:11.340902+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3786, 'log_dups_size': 0, 'ondisk_log_size': 3786, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15488210, 'num_objects': 3783, 'num_object_clones': 0, 'num_object_copies': 7566, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3784, 'num_write_kb': 15127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 256}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:13.540 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:55:13.540 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'3815", 'reported_seq': 3900, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:11.298825+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:55:11.298825+0000', 'last_peered': '2024-09-18T11:55:11.298825+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:55:11.298825+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3815, 'log_dups_size': 0, 'ondisk_log_size': 3815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15602898, 'num_objects': 3811, 'num_object_clones': 0, 'num_object_copies': 7622, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3811, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3813, 'num_write_kb': 15240, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 252}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:13.540 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:55:13.540 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'3765", 'reported_seq': 4285, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:12.459137+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:55:12.459137+0000', 'last_peered': '2024-09-18T11:55:12.459137+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:55:12.459137+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 3765, 'log_dups_size': 0, 'ondisk_log_size': 3765, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15394002, 'num_objects': 3760, 'num_object_clones': 0, 'num_object_copies': 7520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 3294, 'num_objects_unfound': 0, 'num_objects_dirty': 3760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3763, 'num_write_kb': 15036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 186, 'num_bytes_recovered': 753664, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3294}, {'shards': '2', 'objects': 465}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:13.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:55:13.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'3782", 'reported_seq': 3856, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:11.340840+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:55:11.340840+0000', 'last_peered': '2024-09-18T11:55:11.340840+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:55:11.340840+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3782, 'log_dups_size': 0, 'ondisk_log_size': 3782, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15459538, 'num_objects': 3776, 'num_object_clones': 0, 'num_object_copies': 7552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3780, 'num_write_kb': 15100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 244}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:13.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:55:13.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'3850", 'reported_seq': 3904, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:12.461468+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:55:12.461468+0000', 'last_peered': '2024-09-18T11:55:12.461468+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:55:12.461468+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3850, 'log_dups_size': 0, 'ondisk_log_size': 3850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15750354, 'num_objects': 3847, 'num_object_clones': 0, 'num_object_copies': 7694, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3847, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3848, 'num_write_kb': 15383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 297}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:13.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:55:13.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'3895", 'reported_seq': 4102, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:12.402347+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:55:12.402347+0000', 'last_peered': '2024-09-18T11:55:12.402347+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:55:12.402347+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 3895, 'log_dups_size': 0, 'ondisk_log_size': 3895, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15929344, 'num_objects': 3889, 'num_object_clones': 0, 'num_object_copies': 7778, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 375, 'num_write_kb': 1500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 296}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:13.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:55:13.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'3800", 'reported_seq': 3880, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:11.217995+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:55:11.217995+0000', 'last_peered': '2024-09-18T11:55:11.217995+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:55:11.217995+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3800, 'log_dups_size': 0, 'ondisk_log_size': 3800, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15557225, 'num_objects': 3799, 'num_object_clones': 0, 'num_object_copies': 7598, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3799, 'num_write_kb': 15193, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 264}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:13.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:55:13.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'3815", 'reported_seq': 3897, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:11.325439+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:55:11.325439+0000', 'last_peered': '2024-09-18T11:55:11.325439+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:55:11.325439+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3815, 'log_dups_size': 0, 'ondisk_log_size': 3815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15595323, 'num_objects': 3810, 'num_object_clones': 0, 'num_object_copies': 7620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3812, 'num_write_kb': 15232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 288}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:13.541 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:55:13.542 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'3855", 'reported_seq': 3934, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:11.300697+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:55:11.300697+0000', 'last_peered': '2024-09-18T11:55:11.300697+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:55:11.300697+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3855, 'log_dups_size': 0, 'ondisk_log_size': 3855, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15774930, 'num_objects': 3853, 'num_object_clones': 0, 'num_object_copies': 7706, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3853, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3853, 'num_write_kb': 15406, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 260}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:13.542 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:55:13.542 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'3792", 'reported_seq': 3875, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:11.289392+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:55:11.289392+0000', 'last_peered': '2024-09-18T11:55:11.289392+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:55:11.289392+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3792, 'log_dups_size': 0, 'ondisk_log_size': 3792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15485348, 'num_objects': 3784, 'num_object_clones': 0, 'num_object_copies': 7568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3789, 'num_write_kb': 15127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 261}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:13.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:14.104 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 48 20 62485 62465 5.08215 2.29688 0.0264888 0.015363 2024-09-18T11:55:14.358 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:14.358 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/122236 objects degraded (31.979%), 12 pgs degraded","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":114,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":61118,"data_bytes":250261575,"bytes_used":1438797824,"bytes_avail":288471494656,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":122236,"degraded_ratio":0.31979122353480155,"misplaced_objects":3294,"misplaced_total":122236,"misplaced_ratio":0.026947871330868158,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":76888,"recovering_keys_per_sec":0,"num_objects_recovered":183,"num_bytes_recovered":753664,"num_keys_recovered":0,"write_bytes_sec":498001,"read_op_per_sec":0,"write_op_per_sec":121},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T11:55:03.038775+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:55:14.393 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:14.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:15.104 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 49 20 63066 63046 5.02474 2.26953 0.0361697 0.0155378 2024-09-18T11:55:15.106 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:15.119 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:15.130 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-18T11:55:15.041686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":250261575,"num_objects":61118,"num_object_clones":0,"num_object_copies":122236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3294,"num_objects_unfound":0,"num_objects_dirty":61118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45173,"num_write_kb":180566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":762492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61181,"ondisk_log_size":61181,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1405076,"kb_used_data":486588,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281710444,"statfs":{"total":289910292480,"available":288471494656,"internally_reserved":0,"allocated":498266112,"data_stored":497542862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,6,9,5,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19959191,"num_objects":4872,"num_object_clones":0,"num_object_copies":9744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":-24639,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1191,"num_write_kb":4767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.798423"},"pg_stats":[{"pgid":"2.f","version":"37'3845","reported_seq":3914,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.264390+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:11.264390+0000","last_peered":"2024-09-18T11:55:11.264390+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:11.264390+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15749120,"num_objects":3845,"num_object_clones":0,"num_object_copies":7690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3845,"num_write_kb":15380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":246},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'3858","reported_seq":4014,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:12.437938+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:12.437938+0000","last_peered":"2024-09-18T11:55:12.437938+0000","last_clean":"2024-09-18T11:55:12.437938+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:12.437938+0000","last_undegraded":"2024-09-18T11:55:12.437938+0000","last_fullsized":"2024-09-18T11:55:12.437938+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":3858,"log_dups_size":0,"ondisk_log_size":3858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15777792,"num_objects":3852,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1050,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'3829","reported_seq":4012,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:12.450820+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:12.450820+0000","last_peered":"2024-09-18T11:55:12.450820+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:12.450820+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15667200,"num_objects":3825,"num_object_clones":0,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":687,"num_write_kb":2748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":289}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'3780","reported_seq":3806,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:12.468320+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:12.468320+0000","last_peered":"2024-09-18T11:55:12.468320+0000","last_clean":"2024-09-18T11:55:12.468320+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:12.468320+0000","last_undegraded":"2024-09-18T11:55:12.468320+0000","last_fullsized":"2024-09-18T11:55:12.468320+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3780,"log_dups_size":0,"ondisk_log_size":3780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15482880,"num_objects":3780,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":15120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'3778","reported_seq":3967,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:11.307364+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:11.307364+0000","last_peered":"2024-09-18T11:55:11.307364+0000","last_clean":"2024-09-18T11:55:11.307364+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:11.307364+0000","last_undegraded":"2024-09-18T11:55:11.307364+0000","last_fullsized":"2024-09-18T11:55:11.307364+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15446016,"num_objects":3771,"num_object_clones":0,"num_object_copies":7542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":943,"num_write_kb":3772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'3786","reported_seq":3862,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.340902+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:11.340902+0000","last_peered":"2024-09-18T11:55:11.340902+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:11.340902+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15488210,"num_objects":3783,"num_object_clones":0,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":15127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":256},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'3815","reported_seq":3900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.298825+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:11.298825+0000","last_peered":"2024-09-18T11:55:11.298825+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:11.298825+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15602898,"num_objects":3811,"num_object_clones":0,"num_object_copies":7622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3813,"num_write_kb":15240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":252},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'3933","reported_seq":4156,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:11.354071+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:11.354071+0000","last_peered":"2024-09-18T11:55:11.354071+0000","last_clean":"2024-09-18T11:55:11.354071+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:11.354071+0000","last_undegraded":"2024-09-18T11:55:11.354071+0000","last_fullsized":"2024-09-18T11:55:11.354071+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16101376,"num_objects":3931,"num_object_clones":0,"num_object_copies":7862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":250,"num_write_kb":1000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'3765","reported_seq":4285,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:12.459137+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:12.459137+0000","last_peered":"2024-09-18T11:55:12.459137+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:12.459137+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":3765,"log_dups_size":0,"ondisk_log_size":3765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15394002,"num_objects":3760,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3294,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3763,"num_write_kb":15036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3294},{"shards":"2","objects":465}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'3782","reported_seq":3856,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.340840+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:11.340840+0000","last_peered":"2024-09-18T11:55:11.340840+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:11.340840+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15459538,"num_objects":3776,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":15100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":244},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'3850","reported_seq":3904,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:12.461468+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:12.461468+0000","last_peered":"2024-09-18T11:55:12.461468+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:12.461468+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15750354,"num_objects":3847,"num_object_clones":0,"num_object_copies":7694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":15383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'3895","reported_seq":4102,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:12.402347+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:12.402347+0000","last_peered":"2024-09-18T11:55:12.402347+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:12.402347+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":3895,"log_dups_size":0,"ondisk_log_size":3895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15929344,"num_objects":3889,"num_object_clones":0,"num_object_copies":7778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":375,"num_write_kb":1500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":296}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'3800","reported_seq":3880,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.217995+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:11.217995+0000","last_peered":"2024-09-18T11:55:11.217995+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:11.217995+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3800,"log_dups_size":0,"ondisk_log_size":3800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15557225,"num_objects":3799,"num_object_clones":0,"num_object_copies":7598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3799,"num_write_kb":15193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":264},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'3815","reported_seq":3897,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.325439+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:11.325439+0000","last_peered":"2024-09-18T11:55:11.325439+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:11.325439+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15595323,"num_objects":3810,"num_object_clones":0,"num_object_copies":7620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3812,"num_write_kb":15232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":288},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'3855","reported_seq":3934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.300697+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:11.300697+0000","last_peered":"2024-09-18T11:55:11.300697+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:11.300697+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15774930,"num_objects":3853,"num_object_clones":0,"num_object_copies":7706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3853,"num_write_kb":15406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":260},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'3792","reported_seq":3875,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:11.289392+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:11.289392+0000","last_peered":"2024-09-18T11:55:11.289392+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:11.289392+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3792,"log_dups_size":0,"ondisk_log_size":3792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15485348,"num_objects":3784,"num_object_clones":0,"num_object_copies":7568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":15127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":261},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":250261556,"num_objects":61116,"num_object_clones":0,"num_object_copies":122232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3294,"num_objects_unfound":0,"num_objects_dirty":61116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45171,"num_write_kb":180564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":762473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":500342784,"data_stored":500255809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61178,"ondisk_log_size":61178,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051600,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542008,"kb_used_data":235824,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93829832,"statfs":{"total":96636764160,"available":96081747968,"internally_reserved":0,"allocated":241483776,"data_stored":241270620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.183}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]}]},{"osd":1,"up_from":35,"seq":150323855363,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319212,"kb_used_data":13028,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94052628,"statfs":{"total":96636764160,"available":96309891072,"internally_reserved":0,"allocated":13340672,"data_stored":13047356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6680000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543856,"kb_used_data":237736,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93827984,"statfs":{"total":96636764160,"available":96079855616,"internally_reserved":0,"allocated":243441664,"data_stored":243224886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,5,3,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245186560,"data_stored":245183081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13025280,"data_stored":12945263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242130944,"data_stored":242127465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:16.104 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 50 20 63700 63680 4.97377 2.47656 0.0183421 0.0156959 2024-09-18T11:55:17.104 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 51 20 64327 64307 4.92428 2.44922 0.0225622 0.0158556 2024-09-18T11:55:18.104 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 52 20 64935 64915 4.87526 2.375 0.0371758 0.0160156 2024-09-18T11:55:18.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:18.818 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-18T11:55:17.042102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":257794119,"num_objects":62957,"num_object_clones":0,"num_object_copies":125914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3294,"num_objects_unfound":0,"num_objects_dirty":62957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47012,"num_write_kb":187922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":762492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63020,"ondisk_log_size":63020,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1434176,"kb_used_data":503208,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":281681344,"statfs":{"total":289910292480,"available":288441696256,"internally_reserved":0,"allocated":515284992,"data_stored":514561742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,6,11,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27491735,"num_objects":6711,"num_object_clones":0,"num_object_copies":13422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":-24639,"num_objects_unfound":0,"num_objects_dirty":6711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3030,"num_write_kb":12123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002483"},"pg_stats":[{"pgid":"2.f","version":"37'4041","reported_seq":4110,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.358201+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:16.358201+0000","last_peered":"2024-09-18T11:55:16.358201+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:16.358201+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4041,"log_dups_size":0,"ondisk_log_size":4041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16551936,"num_objects":4041,"num_object_clones":0,"num_object_copies":8082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4041,"num_write_kb":16164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":442},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'3858","reported_seq":4014,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:12.437938+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:12.437938+0000","last_peered":"2024-09-18T11:55:12.437938+0000","last_clean":"2024-09-18T11:55:12.437938+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:12.437938+0000","last_undegraded":"2024-09-18T11:55:12.437938+0000","last_fullsized":"2024-09-18T11:55:12.437938+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":3858,"log_dups_size":0,"ondisk_log_size":3858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15777792,"num_objects":3852,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1050,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'3829","reported_seq":4012,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:12.450820+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:12.450820+0000","last_peered":"2024-09-18T11:55:12.450820+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:12.450820+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15667200,"num_objects":3825,"num_object_clones":0,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":687,"num_write_kb":2748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":289}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'3780","reported_seq":3806,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:12.468320+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:12.468320+0000","last_peered":"2024-09-18T11:55:12.468320+0000","last_clean":"2024-09-18T11:55:12.468320+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:12.468320+0000","last_undegraded":"2024-09-18T11:55:12.468320+0000","last_fullsized":"2024-09-18T11:55:12.468320+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3780,"log_dups_size":0,"ondisk_log_size":3780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15482880,"num_objects":3780,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":15120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'3962","reported_seq":4151,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:16.341895+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:16.341895+0000","last_peered":"2024-09-18T11:55:16.341895+0000","last_clean":"2024-09-18T11:55:16.341895+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:16.341895+0000","last_undegraded":"2024-09-18T11:55:16.341895+0000","last_fullsized":"2024-09-18T11:55:16.341895+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16199680,"num_objects":3955,"num_object_clones":0,"num_object_copies":7910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1127,"num_write_kb":4508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'3970","reported_seq":4049,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.323575+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:16.323575+0000","last_peered":"2024-09-18T11:55:16.323575+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:16.323575+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16241874,"num_objects":3967,"num_object_clones":0,"num_object_copies":7934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":15863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":440},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'3996","reported_seq":4079,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.355286+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:16.355286+0000","last_peered":"2024-09-18T11:55:16.355286+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:16.355286+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3996,"log_dups_size":0,"ondisk_log_size":3996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16344274,"num_objects":3992,"num_object_clones":0,"num_object_copies":7984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":15964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":433},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4110","reported_seq":4333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:16.289748+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:16.289748+0000","last_peered":"2024-09-18T11:55:16.289748+0000","last_clean":"2024-09-18T11:55:16.289748+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:16.289748+0000","last_undegraded":"2024-09-18T11:55:16.289748+0000","last_fullsized":"2024-09-18T11:55:16.289748+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16826368,"num_objects":4108,"num_object_clones":0,"num_object_copies":8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":427,"num_write_kb":1708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'3765","reported_seq":4285,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:12.459137+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:12.459137+0000","last_peered":"2024-09-18T11:55:12.459137+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:12.459137+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":3765,"log_dups_size":0,"ondisk_log_size":3765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15394002,"num_objects":3760,"num_object_clones":0,"num_object_copies":7520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3294,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3763,"num_write_kb":15036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3294},{"shards":"2","objects":465}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'3945","reported_seq":4021,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.347724+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:16.347724+0000","last_peered":"2024-09-18T11:55:16.347724+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:16.347724+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16127186,"num_objects":3939,"num_object_clones":0,"num_object_copies":7878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3943,"num_write_kb":15752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":407},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'3850","reported_seq":3904,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:12.461468+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:12.461468+0000","last_peered":"2024-09-18T11:55:12.461468+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:12.461468+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15750354,"num_objects":3847,"num_object_clones":0,"num_object_copies":7694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":15383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'3895","reported_seq":4102,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:12.402347+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:12.402347+0000","last_peered":"2024-09-18T11:55:12.402347+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:12.402347+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":3895,"log_dups_size":0,"ondisk_log_size":3895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15929344,"num_objects":3889,"num_object_clones":0,"num_object_copies":7778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":375,"num_write_kb":1500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":296}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4010","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.322816+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:16.322816+0000","last_peered":"2024-09-18T11:55:16.322816+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:16.322816+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4010,"log_dups_size":0,"ondisk_log_size":4010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16417385,"num_objects":4009,"num_object_clones":0,"num_object_copies":8018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4009,"num_write_kb":16033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":474},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'3988","reported_seq":4070,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.315861+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:16.315861+0000","last_peered":"2024-09-18T11:55:16.315861+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:16.315861+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3988,"log_dups_size":0,"ondisk_log_size":3988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16303931,"num_objects":3983,"num_object_clones":0,"num_object_copies":7966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":15924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":461},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4027","reported_seq":4106,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.332890+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:16.332890+0000","last_peered":"2024-09-18T11:55:16.332890+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:16.332890+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4027,"log_dups_size":0,"ondisk_log_size":4027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16479442,"num_objects":4025,"num_object_clones":0,"num_object_copies":8050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4025,"num_write_kb":16094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":432},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'3991","reported_seq":4074,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.273113+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:16.273113+0000","last_peered":"2024-09-18T11:55:16.273113+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:16.273113+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3991,"log_dups_size":0,"ondisk_log_size":3991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16300452,"num_objects":3983,"num_object_clones":0,"num_object_copies":7966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":15923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":460},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":257794100,"num_objects":62955,"num_object_clones":0,"num_object_copies":125910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3294,"num_objects_unfound":0,"num_objects_dirty":62955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47010,"num_write_kb":187920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":762473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":519340032,"data_stored":519253057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63017,"ondisk_log_size":63017,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051600,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542008,"kb_used_data":235824,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93829832,"statfs":{"total":96636764160,"available":96081747968,"internally_reserved":0,"allocated":241483776,"data_stored":241270620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.183}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]}]},{"osd":1,"up_from":35,"seq":150323855364,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335124,"kb_used_data":20620,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":94036716,"statfs":{"total":96636764160,"available":96293597184,"internally_reserved":0,"allocated":21114880,"data_stored":20821564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557044,"kb_used_data":246764,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93814796,"statfs":{"total":96636764160,"available":96066351104,"internally_reserved":0,"allocated":252686336,"data_stored":252469558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":254349312,"data_stored":254345833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22859776,"data_stored":22779759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242130944,"data_stored":242127465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:18.837 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:19.104 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 53 20 65515 65495 4.82602 2.26562 0.0864573 0.0161759 2024-09-18T11:55:19.490 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:19.499 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:19.520 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-18T11:55:19.042571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262688839,"num_objects":64152,"num_object_clones":0,"num_object_copies":128304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3180,"num_objects_unfound":0,"num_objects_dirty":64152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48207,"num_write_kb":192702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":1229436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64215,"ondisk_log_size":64215,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1444664,"kb_used_data":509536,"kb_used_omap":13,"kb_used_meta":935090,"kb_avail":281670856,"statfs":{"total":289910292480,"available":288430956544,"internally_reserved":0,"allocated":521764864,"data_stored":521041614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":957532690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,8,10,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27467776,"num_objects":6706,"num_object_clones":0,"num_object_copies":13412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10685,"num_objects_misplaced":3180,"num_objects_unfound":0,"num_objects_dirty":6706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":26824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":1220608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002636"},"pg_stats":[{"pgid":"2.f","version":"37'4041","reported_seq":4110,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.358201+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:16.358201+0000","last_peered":"2024-09-18T11:55:16.358201+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:16.358201+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4041,"log_dups_size":0,"ondisk_log_size":4041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16551936,"num_objects":4041,"num_object_clones":0,"num_object_copies":8082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4041,"num_write_kb":16164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":442},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4074","reported_seq":4230,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:17.414333+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:17.414333+0000","last_peered":"2024-09-18T11:55:17.414333+0000","last_clean":"2024-09-18T11:55:17.414333+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:17.414333+0000","last_undegraded":"2024-09-18T11:55:17.414333+0000","last_fullsized":"2024-09-18T11:55:17.414333+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4074,"log_dups_size":0,"ondisk_log_size":4074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16662528,"num_objects":4068,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1266,"num_write_kb":5064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4016","reported_seq":4199,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:17.405882+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:17.405882+0000","last_peered":"2024-09-18T11:55:17.405882+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:17.405882+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4016,"log_dups_size":0,"ondisk_log_size":4016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16433152,"num_objects":4012,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":874,"num_write_kb":3496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":476}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'3975","reported_seq":4002,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:17.379451+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:17.379451+0000","last_peered":"2024-09-18T11:55:17.379451+0000","last_clean":"2024-09-18T11:55:17.379451+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:17.379451+0000","last_undegraded":"2024-09-18T11:55:17.379451+0000","last_fullsized":"2024-09-18T11:55:17.379451+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3975,"log_dups_size":0,"ondisk_log_size":3975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16281600,"num_objects":3975,"num_object_clones":0,"num_object_copies":7950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":15900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'3962","reported_seq":4151,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:16.341895+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:16.341895+0000","last_peered":"2024-09-18T11:55:16.341895+0000","last_clean":"2024-09-18T11:55:16.341895+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:16.341895+0000","last_undegraded":"2024-09-18T11:55:16.341895+0000","last_fullsized":"2024-09-18T11:55:16.341895+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16199680,"num_objects":3955,"num_object_clones":0,"num_object_copies":7910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1127,"num_write_kb":4508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'3970","reported_seq":4049,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.323575+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:16.323575+0000","last_peered":"2024-09-18T11:55:16.323575+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:16.323575+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16241874,"num_objects":3967,"num_object_clones":0,"num_object_copies":7934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":15863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":440},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'3996","reported_seq":4079,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.355286+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:16.355286+0000","last_peered":"2024-09-18T11:55:16.355286+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:16.355286+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3996,"log_dups_size":0,"ondisk_log_size":3996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16344274,"num_objects":3992,"num_object_clones":0,"num_object_copies":7984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":15964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":433},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4110","reported_seq":4333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:16.289748+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:16.289748+0000","last_peered":"2024-09-18T11:55:16.289748+0000","last_clean":"2024-09-18T11:55:16.289748+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:16.289748+0000","last_undegraded":"2024-09-18T11:55:16.289748+0000","last_fullsized":"2024-09-18T11:55:16.289748+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16826368,"num_objects":4108,"num_object_clones":0,"num_object_copies":8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":427,"num_write_kb":1708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'3963","reported_seq":4711,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:17.457684+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:17.457684+0000","last_peered":"2024-09-18T11:55:17.457684+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:17.457684+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":3963,"log_dups_size":0,"ondisk_log_size":3963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16205010,"num_objects":3958,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3180,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3961,"num_write_kb":15828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":300,"num_bytes_recovered":1220608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3180},{"shards":"2","objects":777}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'3945","reported_seq":4021,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.347724+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:16.347724+0000","last_peered":"2024-09-18T11:55:16.347724+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:16.347724+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16127186,"num_objects":3939,"num_object_clones":0,"num_object_copies":7878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3943,"num_write_kb":15752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":407},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4058","reported_seq":4112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:17.442806+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:17.442806+0000","last_peered":"2024-09-18T11:55:17.442806+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:17.442806+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4058,"log_dups_size":0,"ondisk_log_size":4058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16602322,"num_objects":4055,"num_object_clones":0,"num_object_copies":8110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4056,"num_write_kb":16215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":505}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4086","reported_seq":4293,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:17.455835+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:17.455835+0000","last_peered":"2024-09-18T11:55:17.455835+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:17.455835+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4086,"log_dups_size":0,"ondisk_log_size":4086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16711680,"num_objects":4080,"num_object_clones":0,"num_object_copies":8160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":566,"num_write_kb":2264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":487}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4010","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.322816+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:16.322816+0000","last_peered":"2024-09-18T11:55:16.322816+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:16.322816+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4010,"log_dups_size":0,"ondisk_log_size":4010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16417385,"num_objects":4009,"num_object_clones":0,"num_object_copies":8018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4009,"num_write_kb":16033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":474},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'3988","reported_seq":4070,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.315861+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:16.315861+0000","last_peered":"2024-09-18T11:55:16.315861+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:16.315861+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3988,"log_dups_size":0,"ondisk_log_size":3988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16303931,"num_objects":3983,"num_object_clones":0,"num_object_copies":7966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":15924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":461},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4027","reported_seq":4106,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.332890+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:16.332890+0000","last_peered":"2024-09-18T11:55:16.332890+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:16.332890+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4027,"log_dups_size":0,"ondisk_log_size":4027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16479442,"num_objects":4025,"num_object_clones":0,"num_object_copies":8050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4025,"num_write_kb":16094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":432},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'3991","reported_seq":4074,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.273113+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:16.273113+0000","last_peered":"2024-09-18T11:55:16.273113+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:16.273113+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3991,"log_dups_size":0,"ondisk_log_size":3991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16300452,"num_objects":3983,"num_object_clones":0,"num_object_copies":7966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":15923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":460},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":262688820,"num_objects":64150,"num_object_clones":0,"num_object_copies":128300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3180,"num_objects_unfound":0,"num_objects_dirty":64150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48205,"num_write_kb":192700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":1229417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":525705216,"data_stored":525618241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64212,"ondisk_log_size":64212,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051601,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552496,"kb_used_data":242152,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93819344,"statfs":{"total":96636764160,"available":96071008256,"internally_reserved":0,"allocated":247963648,"data_stored":247750492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.219}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1059999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855364,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335124,"kb_used_data":20620,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":94036716,"statfs":{"total":96636764160,"available":96293597184,"internally_reserved":0,"allocated":21114880,"data_stored":20821564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557044,"kb_used_data":246764,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93814796,"statfs":{"total":96636764160,"available":96066351104,"internally_reserved":0,"allocated":252686336,"data_stored":252469558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":254349312,"data_stored":254345833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22859776,"data_stored":22779759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":248496128,"data_stored":248492649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:19.521 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:55:19.521 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'4041", 'reported_seq': 4110, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:16.358201+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:55:16.358201+0000', 'last_peered': '2024-09-18T11:55:16.358201+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:55:16.358201+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4041, 'log_dups_size': 0, 'ondisk_log_size': 4041, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16551936, 'num_objects': 4041, 'num_object_clones': 0, 'num_object_copies': 8082, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4041, 'num_write_kb': 16164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 442}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:19.522 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:55:19.522 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'4016", 'reported_seq': 4199, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:17.405882+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:55:17.405882+0000', 'last_peered': '2024-09-18T11:55:17.405882+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:55:17.405882+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 4016, 'log_dups_size': 0, 'ondisk_log_size': 4016, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16433152, 'num_objects': 4012, 'num_object_clones': 0, 'num_object_copies': 8024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 874, 'num_write_kb': 3496, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 476}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:19.522 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:55:19.522 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'3970", 'reported_seq': 4049, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:16.323575+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:55:16.323575+0000', 'last_peered': '2024-09-18T11:55:16.323575+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:55:16.323575+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3970, 'log_dups_size': 0, 'ondisk_log_size': 3970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16241874, 'num_objects': 3967, 'num_object_clones': 0, 'num_object_copies': 7934, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3968, 'num_write_kb': 15863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 440}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:19.522 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:55:19.522 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'3996", 'reported_seq': 4079, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:16.355286+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:55:16.355286+0000', 'last_peered': '2024-09-18T11:55:16.355286+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:55:16.355286+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3996, 'log_dups_size': 0, 'ondisk_log_size': 3996, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16344274, 'num_objects': 3992, 'num_object_clones': 0, 'num_object_copies': 7984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3994, 'num_write_kb': 15964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 433}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:19.522 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:55:19.522 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'3963", 'reported_seq': 4711, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:17.457684+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:55:17.457684+0000', 'last_peered': '2024-09-18T11:55:17.457684+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:55:17.457684+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 3963, 'log_dups_size': 0, 'ondisk_log_size': 3963, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16205010, 'num_objects': 3958, 'num_object_clones': 0, 'num_object_copies': 7916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 3180, 'num_objects_unfound': 0, 'num_objects_dirty': 3958, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3961, 'num_write_kb': 15828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 300, 'num_bytes_recovered': 1220608, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3180}, {'shards': '2', 'objects': 777}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:19.523 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:55:19.523 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'3945", 'reported_seq': 4021, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:16.347724+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:55:16.347724+0000', 'last_peered': '2024-09-18T11:55:16.347724+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:55:16.347724+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3945, 'log_dups_size': 0, 'ondisk_log_size': 3945, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16127186, 'num_objects': 3939, 'num_object_clones': 0, 'num_object_copies': 7878, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3943, 'num_write_kb': 15752, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 407}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:19.523 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:55:19.523 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'4058", 'reported_seq': 4112, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:17.442806+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:55:17.442806+0000', 'last_peered': '2024-09-18T11:55:17.442806+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:55:17.442806+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4058, 'log_dups_size': 0, 'ondisk_log_size': 4058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16602322, 'num_objects': 4055, 'num_object_clones': 0, 'num_object_copies': 8110, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4056, 'num_write_kb': 16215, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 505}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:19.523 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:55:19.523 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'4086", 'reported_seq': 4293, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:17.455835+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:55:17.455835+0000', 'last_peered': '2024-09-18T11:55:17.455835+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:55:17.455835+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 4086, 'log_dups_size': 0, 'ondisk_log_size': 4086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16711680, 'num_objects': 4080, 'num_object_clones': 0, 'num_object_copies': 8160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 566, 'num_write_kb': 2264, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 487}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:19.523 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:55:19.523 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'4010", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:16.322816+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:55:16.322816+0000', 'last_peered': '2024-09-18T11:55:16.322816+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:55:16.322816+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4010, 'log_dups_size': 0, 'ondisk_log_size': 4010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16417385, 'num_objects': 4009, 'num_object_clones': 0, 'num_object_copies': 8018, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4009, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4009, 'num_write_kb': 16033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 474}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:19.524 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:55:19.524 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'3988", 'reported_seq': 4070, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:16.315861+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:55:16.315861+0000', 'last_peered': '2024-09-18T11:55:16.315861+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:55:16.315861+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3988, 'log_dups_size': 0, 'ondisk_log_size': 3988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16303931, 'num_objects': 3983, 'num_object_clones': 0, 'num_object_copies': 7966, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 15924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 461}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:19.524 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:55:19.524 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'4027", 'reported_seq': 4106, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:16.332890+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:55:16.332890+0000', 'last_peered': '2024-09-18T11:55:16.332890+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:55:16.332890+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4027, 'log_dups_size': 0, 'ondisk_log_size': 4027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16479442, 'num_objects': 4025, 'num_object_clones': 0, 'num_object_copies': 8050, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4025, 'num_write_kb': 16094, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 432}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:19.524 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:55:19.524 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'3991", 'reported_seq': 4074, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:16.273113+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:55:16.273113+0000', 'last_peered': '2024-09-18T11:55:16.273113+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:55:16.273113+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 3991, 'log_dups_size': 0, 'ondisk_log_size': 3991, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16300452, 'num_objects': 3983, 'num_object_clones': 0, 'num_object_copies': 7966, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3988, 'num_write_kb': 15923, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 460}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:19.524 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:20.104 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 54 20 66093 66073 4.77847 2.25781 0.018713 0.0163401 2024-09-18T11:55:20.354 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:20.354 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/128304 objects degraded (30.467%), 12 pgs degraded","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":120,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":64152,"data_bytes":262688839,"bytes_used":1479335936,"bytes_avail":288430956544,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":128304,"degraded_ratio":0.30466704077815188,"misplaced_objects":3180,"misplaced_total":128304,"misplaced_ratio":0.024784885895997012,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":101694,"recovering_keys_per_sec":0,"num_objects_recovered":298,"num_bytes_recovered":1220608,"num_keys_recovered":0,"write_bytes_sec":2288478,"read_op_per_sec":0,"write_op_per_sec":558},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:55:19.042648+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:55:20.392 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:20.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:20.916 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:20.943 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:20.952 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-18T11:55:19.042571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262688839,"num_objects":64152,"num_object_clones":0,"num_object_copies":128304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3180,"num_objects_unfound":0,"num_objects_dirty":64152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48207,"num_write_kb":192702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":1229436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64215,"ondisk_log_size":64215,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1444664,"kb_used_data":509536,"kb_used_omap":13,"kb_used_meta":935090,"kb_avail":281670856,"statfs":{"total":289910292480,"available":288430956544,"internally_reserved":0,"allocated":521764864,"data_stored":521041614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":957532690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,8,10,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27467776,"num_objects":6706,"num_object_clones":0,"num_object_copies":13412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10685,"num_objects_misplaced":3180,"num_objects_unfound":0,"num_objects_dirty":6706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":26824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":1220608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002636"},"pg_stats":[{"pgid":"2.f","version":"37'4041","reported_seq":4110,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.358201+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:16.358201+0000","last_peered":"2024-09-18T11:55:16.358201+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:16.358201+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4041,"log_dups_size":0,"ondisk_log_size":4041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16551936,"num_objects":4041,"num_object_clones":0,"num_object_copies":8082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4041,"num_write_kb":16164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":442},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4074","reported_seq":4230,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:17.414333+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:17.414333+0000","last_peered":"2024-09-18T11:55:17.414333+0000","last_clean":"2024-09-18T11:55:17.414333+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:17.414333+0000","last_undegraded":"2024-09-18T11:55:17.414333+0000","last_fullsized":"2024-09-18T11:55:17.414333+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4074,"log_dups_size":0,"ondisk_log_size":4074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16662528,"num_objects":4068,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1266,"num_write_kb":5064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4016","reported_seq":4199,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:17.405882+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:17.405882+0000","last_peered":"2024-09-18T11:55:17.405882+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:17.405882+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4016,"log_dups_size":0,"ondisk_log_size":4016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16433152,"num_objects":4012,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":874,"num_write_kb":3496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":476}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'3975","reported_seq":4002,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:17.379451+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:17.379451+0000","last_peered":"2024-09-18T11:55:17.379451+0000","last_clean":"2024-09-18T11:55:17.379451+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:17.379451+0000","last_undegraded":"2024-09-18T11:55:17.379451+0000","last_fullsized":"2024-09-18T11:55:17.379451+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3975,"log_dups_size":0,"ondisk_log_size":3975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16281600,"num_objects":3975,"num_object_clones":0,"num_object_copies":7950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":15900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'3962","reported_seq":4151,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:16.341895+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:16.341895+0000","last_peered":"2024-09-18T11:55:16.341895+0000","last_clean":"2024-09-18T11:55:16.341895+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:16.341895+0000","last_undegraded":"2024-09-18T11:55:16.341895+0000","last_fullsized":"2024-09-18T11:55:16.341895+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16199680,"num_objects":3955,"num_object_clones":0,"num_object_copies":7910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1127,"num_write_kb":4508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'3970","reported_seq":4049,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.323575+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:16.323575+0000","last_peered":"2024-09-18T11:55:16.323575+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:16.323575+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16241874,"num_objects":3967,"num_object_clones":0,"num_object_copies":7934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3968,"num_write_kb":15863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":440},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'3996","reported_seq":4079,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.355286+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:16.355286+0000","last_peered":"2024-09-18T11:55:16.355286+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:16.355286+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3996,"log_dups_size":0,"ondisk_log_size":3996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16344274,"num_objects":3992,"num_object_clones":0,"num_object_copies":7984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":15964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":433},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4110","reported_seq":4333,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:16.289748+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:16.289748+0000","last_peered":"2024-09-18T11:55:16.289748+0000","last_clean":"2024-09-18T11:55:16.289748+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:16.289748+0000","last_undegraded":"2024-09-18T11:55:16.289748+0000","last_fullsized":"2024-09-18T11:55:16.289748+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16826368,"num_objects":4108,"num_object_clones":0,"num_object_copies":8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":427,"num_write_kb":1708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'3963","reported_seq":4711,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:17.457684+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:17.457684+0000","last_peered":"2024-09-18T11:55:17.457684+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:17.457684+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":3963,"log_dups_size":0,"ondisk_log_size":3963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16205010,"num_objects":3958,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3180,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3961,"num_write_kb":15828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":300,"num_bytes_recovered":1220608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3180},{"shards":"2","objects":777}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'3945","reported_seq":4021,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.347724+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:16.347724+0000","last_peered":"2024-09-18T11:55:16.347724+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:16.347724+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16127186,"num_objects":3939,"num_object_clones":0,"num_object_copies":7878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3943,"num_write_kb":15752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":407},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4058","reported_seq":4112,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:17.442806+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:17.442806+0000","last_peered":"2024-09-18T11:55:17.442806+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:17.442806+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4058,"log_dups_size":0,"ondisk_log_size":4058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16602322,"num_objects":4055,"num_object_clones":0,"num_object_copies":8110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4056,"num_write_kb":16215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":505}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4086","reported_seq":4293,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:17.455835+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:17.455835+0000","last_peered":"2024-09-18T11:55:17.455835+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:17.455835+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4086,"log_dups_size":0,"ondisk_log_size":4086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16711680,"num_objects":4080,"num_object_clones":0,"num_object_copies":8160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":566,"num_write_kb":2264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":487}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4010","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.322816+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:16.322816+0000","last_peered":"2024-09-18T11:55:16.322816+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:16.322816+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4010,"log_dups_size":0,"ondisk_log_size":4010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16417385,"num_objects":4009,"num_object_clones":0,"num_object_copies":8018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4009,"num_write_kb":16033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":474},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'3988","reported_seq":4070,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.315861+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:16.315861+0000","last_peered":"2024-09-18T11:55:16.315861+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:16.315861+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3988,"log_dups_size":0,"ondisk_log_size":3988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16303931,"num_objects":3983,"num_object_clones":0,"num_object_copies":7966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":15924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":461},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4027","reported_seq":4106,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.332890+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:16.332890+0000","last_peered":"2024-09-18T11:55:16.332890+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:16.332890+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4027,"log_dups_size":0,"ondisk_log_size":4027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16479442,"num_objects":4025,"num_object_clones":0,"num_object_copies":8050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4025,"num_write_kb":16094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":432},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'3991","reported_seq":4074,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:16.273113+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:16.273113+0000","last_peered":"2024-09-18T11:55:16.273113+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:16.273113+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":3991,"log_dups_size":0,"ondisk_log_size":3991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16300452,"num_objects":3983,"num_object_clones":0,"num_object_copies":7966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":15923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":460},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":262688820,"num_objects":64150,"num_object_clones":0,"num_object_copies":128300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3180,"num_objects_unfound":0,"num_objects_dirty":64150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48205,"num_write_kb":192700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":1229417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":525705216,"data_stored":525618241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64212,"ondisk_log_size":64212,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051601,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552496,"kb_used_data":242152,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93819344,"statfs":{"total":96636764160,"available":96071008256,"internally_reserved":0,"allocated":247963648,"data_stored":247750492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.219}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1059999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855364,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335124,"kb_used_data":20620,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":94036716,"statfs":{"total":96636764160,"available":96293597184,"internally_reserved":0,"allocated":21114880,"data_stored":20821564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557044,"kb_used_data":246764,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93814796,"statfs":{"total":96636764160,"available":96066351104,"internally_reserved":0,"allocated":252686336,"data_stored":252469558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":254349312,"data_stored":254345833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22859776,"data_stored":22779759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":248496128,"data_stored":248492649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:21.104 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 55 20 66667 66647 4.73236 2.24219 0.017503 0.0165002 2024-09-18T11:55:22.104 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 56 20 67280 67260 4.69061 2.39453 0.0340636 0.016646 2024-09-18T11:55:23.106 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 57 19 67894 67875 4.65034 2.40234 0.0568952 0.016787 2024-09-18T11:55:23.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:24.106 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 58 20 68463 68443 4.60842 2.21875 0.0306418 0.0169416 2024-09-18T11:55:24.547 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:24.551 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:24.570 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-09-18T11:55:23.043435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274817095,"num_objects":67113,"num_object_clones":0,"num_object_copies":134226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3063,"num_objects_unfound":0,"num_objects_dirty":67113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51168,"num_write_kb":204546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":438,"num_bytes_recovered":1708668,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67176,"ondisk_log_size":67176,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1437520,"kb_used_data":535672,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281678000,"statfs":{"total":289910292480,"available":288438272000,"internally_reserved":0,"allocated":548528128,"data_stored":547804878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,7,4,19,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37179392,"num_objects":9077,"num_object_clones":0,"num_object_copies":18154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-357,"num_objects_unfound":0,"num_objects_dirty":9077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9077,"num_write_kb":36308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":357,"num_bytes_recovered":1462272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002576"},"pg_stats":[{"pgid":"2.f","version":"37'4250","reported_seq":4319,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.328737+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:21.328737+0000","last_peered":"2024-09-18T11:55:21.328737+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:21.328737+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17408000,"num_objects":4250,"num_object_clones":0,"num_object_copies":8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4250,"num_write_kb":17000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":651},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4268","reported_seq":4424,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:22.466031+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:22.466031+0000","last_peered":"2024-09-18T11:55:22.466031+0000","last_clean":"2024-09-18T11:55:22.466031+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:22.466031+0000","last_undegraded":"2024-09-18T11:55:22.466031+0000","last_fullsized":"2024-09-18T11:55:22.466031+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4268,"log_dups_size":0,"ondisk_log_size":4268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17457152,"num_objects":4262,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":5840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4224","reported_seq":4407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:22.490711+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:22.490711+0000","last_peered":"2024-09-18T11:55:22.490711+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:22.490711+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17285120,"num_objects":4220,"num_object_clones":0,"num_object_copies":8440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1082,"num_write_kb":4328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":684}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4185","reported_seq":4212,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:22.470075+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:22.470075+0000","last_peered":"2024-09-18T11:55:22.470075+0000","last_clean":"2024-09-18T11:55:22.470075+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:22.470075+0000","last_undegraded":"2024-09-18T11:55:22.470075+0000","last_fullsized":"2024-09-18T11:55:22.470075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17141760,"num_objects":4185,"num_object_clones":0,"num_object_copies":8370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":16740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'4134","reported_seq":4323,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:21.373729+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:21.373729+0000","last_peered":"2024-09-18T11:55:21.373729+0000","last_clean":"2024-09-18T11:55:21.373729+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:21.373729+0000","last_undegraded":"2024-09-18T11:55:21.373729+0000","last_fullsized":"2024-09-18T11:55:21.373729+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":4134,"log_dups_size":0,"ondisk_log_size":4134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16904192,"num_objects":4127,"num_object_clones":0,"num_object_copies":8254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1299,"num_write_kb":5196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4129","reported_seq":4208,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.334983+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:21.334983+0000","last_peered":"2024-09-18T11:55:21.334983+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:21.334983+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16893138,"num_objects":4126,"num_object_clones":0,"num_object_copies":8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":16499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":599},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'4170","reported_seq":4253,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.374936+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:21.374936+0000","last_peered":"2024-09-18T11:55:21.374936+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:21.374936+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4170,"log_dups_size":0,"ondisk_log_size":4170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17056978,"num_objects":4166,"num_object_clones":0,"num_object_copies":8332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4168,"num_write_kb":16660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":607},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4273","reported_seq":4496,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:21.199741+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:21.199741+0000","last_peered":"2024-09-18T11:55:21.199741+0000","last_clean":"2024-09-18T11:55:21.199741+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:21.199741+0000","last_undegraded":"2024-09-18T11:55:21.199741+0000","last_fullsized":"2024-09-18T11:55:21.199741+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17494016,"num_objects":4271,"num_object_clones":0,"num_object_copies":8542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":590,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4148","reported_seq":5129,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:22.474935+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:22.474935+0000","last_peered":"2024-09-18T11:55:22.474935+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:22.474935+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4148,"log_dups_size":0,"ondisk_log_size":4148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16962770,"num_objects":4143,"num_object_clones":0,"num_object_copies":8286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3063,"num_objects_unfound":0,"num_objects_dirty":4143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4146,"num_write_kb":16568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":1699840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3063},{"shards":"2","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'4139","reported_seq":4215,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.344222+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:21.344222+0000","last_peered":"2024-09-18T11:55:21.344222+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:21.344222+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4139,"log_dups_size":0,"ondisk_log_size":4139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16921810,"num_objects":4133,"num_object_clones":0,"num_object_copies":8266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4137,"num_write_kb":16528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":601},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4238","reported_seq":4292,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:22.419867+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:22.419867+0000","last_peered":"2024-09-18T11:55:22.419867+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:22.419867+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17339602,"num_objects":4235,"num_object_clones":0,"num_object_copies":8470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4236,"num_write_kb":16935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":685}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4265","reported_seq":4472,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:22.447063+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:22.447063+0000","last_peered":"2024-09-18T11:55:22.447063+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:22.447063+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17444864,"num_objects":4259,"num_object_clones":0,"num_object_copies":8518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":745,"num_write_kb":2980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4197","reported_seq":4277,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.325425+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:21.325425+0000","last_peered":"2024-09-18T11:55:21.325425+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:21.325425+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17183337,"num_objects":4196,"num_object_clones":0,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4196,"num_write_kb":16781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":661},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'4176","reported_seq":4258,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.363454+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:21.363454+0000","last_peered":"2024-09-18T11:55:21.363454+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:21.363454+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4176,"log_dups_size":0,"ondisk_log_size":4176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17073979,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4173,"num_write_kb":16676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":649},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4191","reported_seq":4270,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.301472+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:21.301472+0000","last_peered":"2024-09-18T11:55:21.301472+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:21.301472+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4191,"log_dups_size":0,"ondisk_log_size":4191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17151186,"num_objects":4189,"num_object_clones":0,"num_object_copies":8378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4189,"num_write_kb":16750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":596},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'4186","reported_seq":4269,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.301462+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:21.301462+0000","last_peered":"2024-09-18T11:55:21.301462+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:21.301462+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17099172,"num_objects":4178,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":16703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":655},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":274817076,"num_objects":67111,"num_object_clones":0,"num_object_copies":134222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3063,"num_objects_unfound":0,"num_objects_dirty":67111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51166,"num_write_kb":204544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":1708649,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":550625280,"data_stored":550538305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67173,"ondisk_log_size":67173,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051602,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545900,"kb_used_data":248036,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93825940,"statfs":{"total":96636764160,"available":96077762560,"internally_reserved":0,"allocated":253988864,"data_stored":253775708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":3.7730000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.214}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]},{"osd":1,"up_from":35,"seq":150323855365,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332272,"kb_used_data":30248,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94039568,"statfs":{"total":96636764160,"available":96296517632,"internally_reserved":0,"allocated":30973952,"data_stored":30680636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559348,"kb_used_data":257388,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93812492,"statfs":{"total":96636764160,"available":96063991808,"internally_reserved":0,"allocated":263565312,"data_stored":263348534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.185},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":1.603}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":263446528,"data_stored":263443049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32526336,"data_stored":32446319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254652416,"data_stored":254648937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:24.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:25.106 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 59 20 69019 68999 4.56713 2.17188 0.0114347 0.0170903 2024-09-18T11:55:25.220 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:25.234 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:25.253 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-18T11:55:25.043917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274817095,"num_objects":67113,"num_object_clones":0,"num_object_copies":134226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3063,"num_objects_unfound":0,"num_objects_dirty":67113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51168,"num_write_kb":204546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":438,"num_bytes_recovered":1708668,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67176,"ondisk_log_size":67176,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1437520,"kb_used_data":535672,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281678000,"statfs":{"total":289910292480,"available":288438272000,"internally_reserved":0,"allocated":548528128,"data_stored":547804878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,7,4,19,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24555520,"num_objects":5995,"num_object_clones":0,"num_object_copies":11990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-231,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5995,"num_write_kb":23980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":946176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002664"},"pg_stats":[{"pgid":"2.f","version":"37'4250","reported_seq":4319,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.328737+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:21.328737+0000","last_peered":"2024-09-18T11:55:21.328737+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:21.328737+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17408000,"num_objects":4250,"num_object_clones":0,"num_object_copies":8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4250,"num_write_kb":17000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":651},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4268","reported_seq":4424,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:22.466031+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:22.466031+0000","last_peered":"2024-09-18T11:55:22.466031+0000","last_clean":"2024-09-18T11:55:22.466031+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:22.466031+0000","last_undegraded":"2024-09-18T11:55:22.466031+0000","last_fullsized":"2024-09-18T11:55:22.466031+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4268,"log_dups_size":0,"ondisk_log_size":4268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17457152,"num_objects":4262,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":5840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4224","reported_seq":4407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:22.490711+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:22.490711+0000","last_peered":"2024-09-18T11:55:22.490711+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:22.490711+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17285120,"num_objects":4220,"num_object_clones":0,"num_object_copies":8440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1082,"num_write_kb":4328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":684}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4185","reported_seq":4212,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:22.470075+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:22.470075+0000","last_peered":"2024-09-18T11:55:22.470075+0000","last_clean":"2024-09-18T11:55:22.470075+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:22.470075+0000","last_undegraded":"2024-09-18T11:55:22.470075+0000","last_fullsized":"2024-09-18T11:55:22.470075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17141760,"num_objects":4185,"num_object_clones":0,"num_object_copies":8370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":16740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'4134","reported_seq":4323,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:21.373729+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:21.373729+0000","last_peered":"2024-09-18T11:55:21.373729+0000","last_clean":"2024-09-18T11:55:21.373729+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:21.373729+0000","last_undegraded":"2024-09-18T11:55:21.373729+0000","last_fullsized":"2024-09-18T11:55:21.373729+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":4134,"log_dups_size":0,"ondisk_log_size":4134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16904192,"num_objects":4127,"num_object_clones":0,"num_object_copies":8254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1299,"num_write_kb":5196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4129","reported_seq":4208,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.334983+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:21.334983+0000","last_peered":"2024-09-18T11:55:21.334983+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:21.334983+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16893138,"num_objects":4126,"num_object_clones":0,"num_object_copies":8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":16499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":599},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'4170","reported_seq":4253,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.374936+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:21.374936+0000","last_peered":"2024-09-18T11:55:21.374936+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:21.374936+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4170,"log_dups_size":0,"ondisk_log_size":4170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17056978,"num_objects":4166,"num_object_clones":0,"num_object_copies":8332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4168,"num_write_kb":16660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":607},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4273","reported_seq":4496,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:21.199741+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:21.199741+0000","last_peered":"2024-09-18T11:55:21.199741+0000","last_clean":"2024-09-18T11:55:21.199741+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:21.199741+0000","last_undegraded":"2024-09-18T11:55:21.199741+0000","last_fullsized":"2024-09-18T11:55:21.199741+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17494016,"num_objects":4271,"num_object_clones":0,"num_object_copies":8542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":590,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4148","reported_seq":5129,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:22.474935+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:22.474935+0000","last_peered":"2024-09-18T11:55:22.474935+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:22.474935+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4148,"log_dups_size":0,"ondisk_log_size":4148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16962770,"num_objects":4143,"num_object_clones":0,"num_object_copies":8286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3063,"num_objects_unfound":0,"num_objects_dirty":4143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4146,"num_write_kb":16568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":1699840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3063},{"shards":"2","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'4139","reported_seq":4215,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.344222+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:21.344222+0000","last_peered":"2024-09-18T11:55:21.344222+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:21.344222+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4139,"log_dups_size":0,"ondisk_log_size":4139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16921810,"num_objects":4133,"num_object_clones":0,"num_object_copies":8266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4137,"num_write_kb":16528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":601},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4238","reported_seq":4292,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:22.419867+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:22.419867+0000","last_peered":"2024-09-18T11:55:22.419867+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:22.419867+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17339602,"num_objects":4235,"num_object_clones":0,"num_object_copies":8470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4236,"num_write_kb":16935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":685}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4265","reported_seq":4472,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:22.447063+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:22.447063+0000","last_peered":"2024-09-18T11:55:22.447063+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:22.447063+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17444864,"num_objects":4259,"num_object_clones":0,"num_object_copies":8518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":745,"num_write_kb":2980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4197","reported_seq":4277,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.325425+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:21.325425+0000","last_peered":"2024-09-18T11:55:21.325425+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:21.325425+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17183337,"num_objects":4196,"num_object_clones":0,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4196,"num_write_kb":16781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":661},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'4176","reported_seq":4258,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.363454+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:21.363454+0000","last_peered":"2024-09-18T11:55:21.363454+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:21.363454+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4176,"log_dups_size":0,"ondisk_log_size":4176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17073979,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4173,"num_write_kb":16676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":649},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4191","reported_seq":4270,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.301472+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:21.301472+0000","last_peered":"2024-09-18T11:55:21.301472+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:21.301472+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4191,"log_dups_size":0,"ondisk_log_size":4191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17151186,"num_objects":4189,"num_object_clones":0,"num_object_copies":8378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4189,"num_write_kb":16750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":596},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'4186","reported_seq":4269,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.301462+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:21.301462+0000","last_peered":"2024-09-18T11:55:21.301462+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:21.301462+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17099172,"num_objects":4178,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":16703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":655},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":274817076,"num_objects":67111,"num_object_clones":0,"num_object_copies":134222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3063,"num_objects_unfound":0,"num_objects_dirty":67111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51166,"num_write_kb":204544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":1708649,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":550625280,"data_stored":550538305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67173,"ondisk_log_size":67173,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051602,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545900,"kb_used_data":248036,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93825940,"statfs":{"total":96636764160,"available":96077762560,"internally_reserved":0,"allocated":253988864,"data_stored":253775708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":3.7730000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.214}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]},{"osd":1,"up_from":35,"seq":150323855365,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332272,"kb_used_data":30248,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94039568,"statfs":{"total":96636764160,"available":96296517632,"internally_reserved":0,"allocated":30973952,"data_stored":30680636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559348,"kb_used_data":257388,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93812492,"statfs":{"total":96636764160,"available":96063991808,"internally_reserved":0,"allocated":263565312,"data_stored":263348534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.185},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":1.603}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":263446528,"data_stored":263443049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32526336,"data_stored":32446319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254652416,"data_stored":254648937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:25.255 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:55:25.255 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'4250", 'reported_seq': 4319, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:21.328737+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:55:21.328737+0000', 'last_peered': '2024-09-18T11:55:21.328737+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:55:21.328737+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4250, 'log_dups_size': 0, 'ondisk_log_size': 4250, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17408000, 'num_objects': 4250, 'num_object_clones': 0, 'num_object_copies': 8500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4250, 'num_write_kb': 17000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 651}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:25.255 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:55:25.255 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'4224", 'reported_seq': 4407, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:22.490711+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:55:22.490711+0000', 'last_peered': '2024-09-18T11:55:22.490711+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:55:22.490711+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 4224, 'log_dups_size': 0, 'ondisk_log_size': 4224, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17285120, 'num_objects': 4220, 'num_object_clones': 0, 'num_object_copies': 8440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1082, 'num_write_kb': 4328, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 684}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:25.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:55:25.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'4129", 'reported_seq': 4208, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:21.334983+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:55:21.334983+0000', 'last_peered': '2024-09-18T11:55:21.334983+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:55:21.334983+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4129, 'log_dups_size': 0, 'ondisk_log_size': 4129, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16893138, 'num_objects': 4126, 'num_object_clones': 0, 'num_object_copies': 8252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4127, 'num_write_kb': 16499, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 599}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:25.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:55:25.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'4170", 'reported_seq': 4253, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:21.374936+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:55:21.374936+0000', 'last_peered': '2024-09-18T11:55:21.374936+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:55:21.374936+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4170, 'log_dups_size': 0, 'ondisk_log_size': 4170, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17056978, 'num_objects': 4166, 'num_object_clones': 0, 'num_object_copies': 8332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4168, 'num_write_kb': 16660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 607}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:25.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:55:25.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'4148", 'reported_seq': 5129, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:22.474935+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:55:22.474935+0000', 'last_peered': '2024-09-18T11:55:22.474935+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:55:22.474935+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 4148, 'log_dups_size': 0, 'ondisk_log_size': 4148, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16962770, 'num_objects': 4143, 'num_object_clones': 0, 'num_object_copies': 8286, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 3063, 'num_objects_unfound': 0, 'num_objects_dirty': 4143, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4146, 'num_write_kb': 16568, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 417, 'num_bytes_recovered': 1699840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3063}, {'shards': '2', 'objects': 1079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:25.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:55:25.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'4139", 'reported_seq': 4215, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:21.344222+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:55:21.344222+0000', 'last_peered': '2024-09-18T11:55:21.344222+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:55:21.344222+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4139, 'log_dups_size': 0, 'ondisk_log_size': 4139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16921810, 'num_objects': 4133, 'num_object_clones': 0, 'num_object_copies': 8266, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4137, 'num_write_kb': 16528, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 601}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:25.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:55:25.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'4238", 'reported_seq': 4292, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:22.419867+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:55:22.419867+0000', 'last_peered': '2024-09-18T11:55:22.419867+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:55:22.419867+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4238, 'log_dups_size': 0, 'ondisk_log_size': 4238, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17339602, 'num_objects': 4235, 'num_object_clones': 0, 'num_object_copies': 8470, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4236, 'num_write_kb': 16935, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 685}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:25.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:55:25.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'4265", 'reported_seq': 4472, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:22.447063+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:55:22.447063+0000', 'last_peered': '2024-09-18T11:55:22.447063+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:55:22.447063+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 4265, 'log_dups_size': 0, 'ondisk_log_size': 4265, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17444864, 'num_objects': 4259, 'num_object_clones': 0, 'num_object_copies': 8518, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 745, 'num_write_kb': 2980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 666}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:25.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:55:25.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'4197", 'reported_seq': 4277, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:21.325425+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:55:21.325425+0000', 'last_peered': '2024-09-18T11:55:21.325425+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:55:21.325425+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4197, 'log_dups_size': 0, 'ondisk_log_size': 4197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17183337, 'num_objects': 4196, 'num_object_clones': 0, 'num_object_copies': 8392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4196, 'num_write_kb': 16781, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 661}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:25.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:55:25.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'4176", 'reported_seq': 4258, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:21.363454+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:55:21.363454+0000', 'last_peered': '2024-09-18T11:55:21.363454+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:55:21.363454+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4176, 'log_dups_size': 0, 'ondisk_log_size': 4176, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17073979, 'num_objects': 4171, 'num_object_clones': 0, 'num_object_copies': 8342, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4173, 'num_write_kb': 16676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 649}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:25.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:55:25.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'4191", 'reported_seq': 4270, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:21.301472+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:55:21.301472+0000', 'last_peered': '2024-09-18T11:55:21.301472+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:55:21.301472+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4191, 'log_dups_size': 0, 'ondisk_log_size': 4191, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17151186, 'num_objects': 4189, 'num_object_clones': 0, 'num_object_copies': 8378, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4189, 'num_write_kb': 16750, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 596}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:25.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:55:25.257 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'4186", 'reported_seq': 4269, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:21.301462+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:55:21.301462+0000', 'last_peered': '2024-09-18T11:55:21.301462+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:55:21.301462+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4186, 'log_dups_size': 0, 'ondisk_log_size': 4186, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17099172, 'num_objects': 4178, 'num_object_clones': 0, 'num_object_copies': 8356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4183, 'num_write_kb': 16703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 655}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:25.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:25.987 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:25.987 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/134226 objects degraded (29.123%), 12 pgs degraded","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":125,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":67113,"data_bytes":274817095,"bytes_used":1472020480,"bytes_avail":288438272000,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":134226,"degraded_ratio":0.29122524697152563,"misplaced_objects":3063,"misplaced_total":134226,"misplaced_ratio":0.022819721961467972,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":78830,"recovering_keys_per_sec":0,"num_objects_recovered":231,"num_bytes_recovered":946176,"num_keys_recovered":0,"write_bytes_sec":2045839,"read_op_per_sec":0,"write_op_per_sec":499},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:55:19.042648+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:55:26.020 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:26.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:26.106 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:2024-09-18T11:55:26.106851+0000 min lat: 0.0021114 max lat: 2.57663 avg lat: 0.0172503 2024-09-18T11:55:26.106 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:55:26.106 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 60 20 69569 69549 4.52682 2.14844 0.0329821 0.0172503 2024-09-18T11:55:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:26.779 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:26.791 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-18T11:55:25.043917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274817095,"num_objects":67113,"num_object_clones":0,"num_object_copies":134226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3063,"num_objects_unfound":0,"num_objects_dirty":67113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51168,"num_write_kb":204546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":438,"num_bytes_recovered":1708668,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67176,"ondisk_log_size":67176,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1437520,"kb_used_data":535672,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281678000,"statfs":{"total":289910292480,"available":288438272000,"internally_reserved":0,"allocated":548528128,"data_stored":547804878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,7,4,19,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24555520,"num_objects":5995,"num_object_clones":0,"num_object_copies":11990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-231,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5995,"num_write_kb":23980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":946176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002664"},"pg_stats":[{"pgid":"2.f","version":"37'4250","reported_seq":4319,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.328737+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:21.328737+0000","last_peered":"2024-09-18T11:55:21.328737+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:21.328737+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17408000,"num_objects":4250,"num_object_clones":0,"num_object_copies":8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4250,"num_write_kb":17000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":651},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4268","reported_seq":4424,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:22.466031+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:22.466031+0000","last_peered":"2024-09-18T11:55:22.466031+0000","last_clean":"2024-09-18T11:55:22.466031+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:22.466031+0000","last_undegraded":"2024-09-18T11:55:22.466031+0000","last_fullsized":"2024-09-18T11:55:22.466031+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4268,"log_dups_size":0,"ondisk_log_size":4268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17457152,"num_objects":4262,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":5840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4224","reported_seq":4407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:22.490711+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:22.490711+0000","last_peered":"2024-09-18T11:55:22.490711+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:22.490711+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17285120,"num_objects":4220,"num_object_clones":0,"num_object_copies":8440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1082,"num_write_kb":4328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":684}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4185","reported_seq":4212,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:22.470075+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:22.470075+0000","last_peered":"2024-09-18T11:55:22.470075+0000","last_clean":"2024-09-18T11:55:22.470075+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:22.470075+0000","last_undegraded":"2024-09-18T11:55:22.470075+0000","last_fullsized":"2024-09-18T11:55:22.470075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17141760,"num_objects":4185,"num_object_clones":0,"num_object_copies":8370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":16740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'4134","reported_seq":4323,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:21.373729+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:21.373729+0000","last_peered":"2024-09-18T11:55:21.373729+0000","last_clean":"2024-09-18T11:55:21.373729+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:21.373729+0000","last_undegraded":"2024-09-18T11:55:21.373729+0000","last_fullsized":"2024-09-18T11:55:21.373729+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":4134,"log_dups_size":0,"ondisk_log_size":4134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16904192,"num_objects":4127,"num_object_clones":0,"num_object_copies":8254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1299,"num_write_kb":5196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4129","reported_seq":4208,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.334983+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:21.334983+0000","last_peered":"2024-09-18T11:55:21.334983+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:21.334983+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16893138,"num_objects":4126,"num_object_clones":0,"num_object_copies":8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":16499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":599},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'4170","reported_seq":4253,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.374936+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:21.374936+0000","last_peered":"2024-09-18T11:55:21.374936+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:21.374936+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4170,"log_dups_size":0,"ondisk_log_size":4170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17056978,"num_objects":4166,"num_object_clones":0,"num_object_copies":8332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4168,"num_write_kb":16660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":607},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4273","reported_seq":4496,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:21.199741+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:21.199741+0000","last_peered":"2024-09-18T11:55:21.199741+0000","last_clean":"2024-09-18T11:55:21.199741+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:21.199741+0000","last_undegraded":"2024-09-18T11:55:21.199741+0000","last_fullsized":"2024-09-18T11:55:21.199741+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17494016,"num_objects":4271,"num_object_clones":0,"num_object_copies":8542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":590,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4148","reported_seq":5129,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:22.474935+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:22.474935+0000","last_peered":"2024-09-18T11:55:22.474935+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:22.474935+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4148,"log_dups_size":0,"ondisk_log_size":4148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16962770,"num_objects":4143,"num_object_clones":0,"num_object_copies":8286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":3063,"num_objects_unfound":0,"num_objects_dirty":4143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4146,"num_write_kb":16568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":1699840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3063},{"shards":"2","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'4139","reported_seq":4215,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.344222+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:21.344222+0000","last_peered":"2024-09-18T11:55:21.344222+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:21.344222+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4139,"log_dups_size":0,"ondisk_log_size":4139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16921810,"num_objects":4133,"num_object_clones":0,"num_object_copies":8266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4137,"num_write_kb":16528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":601},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4238","reported_seq":4292,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:22.419867+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:22.419867+0000","last_peered":"2024-09-18T11:55:22.419867+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:22.419867+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17339602,"num_objects":4235,"num_object_clones":0,"num_object_copies":8470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4236,"num_write_kb":16935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":685}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4265","reported_seq":4472,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:22.447063+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:22.447063+0000","last_peered":"2024-09-18T11:55:22.447063+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:22.447063+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17444864,"num_objects":4259,"num_object_clones":0,"num_object_copies":8518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":745,"num_write_kb":2980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4197","reported_seq":4277,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.325425+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:21.325425+0000","last_peered":"2024-09-18T11:55:21.325425+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:21.325425+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17183337,"num_objects":4196,"num_object_clones":0,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4196,"num_write_kb":16781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":661},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'4176","reported_seq":4258,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.363454+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:21.363454+0000","last_peered":"2024-09-18T11:55:21.363454+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:21.363454+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4176,"log_dups_size":0,"ondisk_log_size":4176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17073979,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4173,"num_write_kb":16676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":649},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4191","reported_seq":4270,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.301472+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:21.301472+0000","last_peered":"2024-09-18T11:55:21.301472+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:21.301472+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4191,"log_dups_size":0,"ondisk_log_size":4191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17151186,"num_objects":4189,"num_object_clones":0,"num_object_copies":8378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4189,"num_write_kb":16750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":596},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'4186","reported_seq":4269,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:21.301462+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:21.301462+0000","last_peered":"2024-09-18T11:55:21.301462+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:21.301462+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17099172,"num_objects":4178,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":16703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":655},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":274817076,"num_objects":67111,"num_object_clones":0,"num_object_copies":134222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":3063,"num_objects_unfound":0,"num_objects_dirty":67111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51166,"num_write_kb":204544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":1708649,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":550625280,"data_stored":550538305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67173,"ondisk_log_size":67173,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051602,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545900,"kb_used_data":248036,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93825940,"statfs":{"total":96636764160,"available":96077762560,"internally_reserved":0,"allocated":253988864,"data_stored":253775708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":3.7730000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.214}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]},{"osd":1,"up_from":35,"seq":150323855365,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332272,"kb_used_data":30248,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94039568,"statfs":{"total":96636764160,"available":96296517632,"internally_reserved":0,"allocated":30973952,"data_stored":30680636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559348,"kb_used_data":257388,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93812492,"statfs":{"total":96636764160,"available":96063991808,"internally_reserved":0,"allocated":263565312,"data_stored":263348534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.185},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":1.603}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":263446528,"data_stored":263443049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32526336,"data_stored":32446319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254652416,"data_stored":254648937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:27.106 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 61 20 70150 70130 4.48982 2.26953 0.0170658 0.0173923 2024-09-18T11:55:28.106 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 62 20 70750 70730 4.4552 2.34375 0.0374647 0.0175264 2024-09-18T11:55:29.106 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 63 20 71360 71340 4.42231 2.38281 0.0125268 0.0176578 2024-09-18T11:55:29.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:30.106 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 64 20 71929 71909 4.38794 2.22266 0.0133143 0.0177919 2024-09-18T11:55:30.561 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:30.579 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:30.602 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-09-18T11:55:29.045002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":286580807,"num_objects":69985,"num_object_clones":0,"num_object_copies":139970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2938,"num_objects_unfound":0,"num_objects_dirty":69985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54040,"num_write_kb":216034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":2220668,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70048,"ondisk_log_size":70048,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1486136,"kb_used_data":559328,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281629384,"statfs":{"total":289910292480,"available":288388489216,"internally_reserved":0,"allocated":572751872,"data_stored":572028622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,10,2,5,15,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28786688,"num_objects":7028,"num_object_clones":0,"num_object_copies":14056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-356,"num_objects_unfound":0,"num_objects_dirty":7028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7028,"num_write_kb":28112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":356,"num_bytes_recovered":1458176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002900"},"pg_stats":[{"pgid":"2.f","version":"37'4437","reported_seq":4505,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.358096+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:26.358096+0000","last_peered":"2024-09-18T11:55:26.358096+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:26.358096+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18173952,"num_objects":4437,"num_object_clones":0,"num_object_copies":8874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4437,"num_write_kb":17748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":838},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4440","reported_seq":4596,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:27.492651+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:27.492651+0000","last_peered":"2024-09-18T11:55:27.492651+0000","last_clean":"2024-09-18T11:55:27.492651+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:27.492651+0000","last_undegraded":"2024-09-18T11:55:27.492651+0000","last_fullsized":"2024-09-18T11:55:27.492651+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18161664,"num_objects":4434,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1632,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4394","reported_seq":4571,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:27.475892+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:27.475892+0000","last_peered":"2024-09-18T11:55:27.475892+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:27.475892+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17981440,"num_objects":4390,"num_object_clones":0,"num_object_copies":8780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1252,"num_write_kb":5008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4358","reported_seq":4385,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:27.486995+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:27.486995+0000","last_peered":"2024-09-18T11:55:27.486995+0000","last_clean":"2024-09-18T11:55:27.486995+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:27.486995+0000","last_undegraded":"2024-09-18T11:55:27.486995+0000","last_fullsized":"2024-09-18T11:55:27.486995+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17850368,"num_objects":4358,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4358,"num_write_kb":17432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'4295","reported_seq":4484,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:26.335966+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:26.335966+0000","last_peered":"2024-09-18T11:55:26.335966+0000","last_clean":"2024-09-18T11:55:26.335966+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:26.335966+0000","last_undegraded":"2024-09-18T11:55:26.335966+0000","last_fullsized":"2024-09-18T11:55:26.335966+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":4295,"log_dups_size":0,"ondisk_log_size":4295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563648,"num_objects":4288,"num_object_clones":0,"num_object_copies":8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":5840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4304","reported_seq":4383,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.343714+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:26.343714+0000","last_peered":"2024-09-18T11:55:26.343714+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:26.343714+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17609938,"num_objects":4301,"num_object_clones":0,"num_object_copies":8602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":17199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":774},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'4348","reported_seq":4433,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.335805+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:26.335805+0000","last_peered":"2024-09-18T11:55:26.335805+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:26.335805+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4348,"log_dups_size":0,"ondisk_log_size":4348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17786066,"num_objects":4344,"num_object_clones":0,"num_object_copies":8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4346,"num_write_kb":17372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":785},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4462","reported_seq":4685,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:26.384506+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:26.384506+0000","last_peered":"2024-09-18T11:55:26.384506+0000","last_clean":"2024-09-18T11:55:26.384506+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:26.384506+0000","last_undegraded":"2024-09-18T11:55:26.384506+0000","last_fullsized":"2024-09-18T11:55:26.384506+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18268160,"num_objects":4460,"num_object_clones":0,"num_object_copies":8920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":779,"num_write_kb":3116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4323","reported_seq":5555,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:27.508559+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:27.508559+0000","last_peered":"2024-09-18T11:55:27.508559+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:27.508559+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17679570,"num_objects":4318,"num_object_clones":0,"num_object_copies":8636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2938,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":17268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":542,"num_bytes_recovered":2211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2938},{"shards":"2","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'4334","reported_seq":4410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.394335+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:26.394335+0000","last_peered":"2024-09-18T11:55:26.394335+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:26.394335+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4334,"log_dups_size":0,"ondisk_log_size":4334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17720530,"num_objects":4328,"num_object_clones":0,"num_object_copies":8656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4332,"num_write_kb":17308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":796},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4403","reported_seq":4457,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:27.510167+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:27.510167+0000","last_peered":"2024-09-18T11:55:27.510167+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:27.510167+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4403,"log_dups_size":0,"ondisk_log_size":4403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18015442,"num_objects":4400,"num_object_clones":0,"num_object_copies":8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4401,"num_write_kb":17595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":850}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4458","reported_seq":4665,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:27.437607+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:27.437607+0000","last_peered":"2024-09-18T11:55:27.437607+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:27.437607+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4458,"log_dups_size":0,"ondisk_log_size":4458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18235392,"num_objects":4452,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":938,"num_write_kb":3752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":859}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4366","reported_seq":4446,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.355516+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:26.355516+0000","last_peered":"2024-09-18T11:55:26.355516+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:26.355516+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17875561,"num_objects":4365,"num_object_clones":0,"num_object_copies":8730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4365,"num_write_kb":17457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":830},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'4369","reported_seq":4451,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.405171+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:26.405171+0000","last_peered":"2024-09-18T11:55:26.405171+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:26.405171+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17864507,"num_objects":4364,"num_object_clones":0,"num_object_copies":8728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4366,"num_write_kb":17448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":842},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4377","reported_seq":4456,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.319560+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:26.319560+0000","last_peered":"2024-09-18T11:55:26.319560+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:26.319560+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17913042,"num_objects":4375,"num_object_clones":0,"num_object_copies":8750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4375,"num_write_kb":17494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":782},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'4377","reported_seq":4460,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.351018+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:26.351018+0000","last_peered":"2024-09-18T11:55:26.351018+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:26.351018+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17881508,"num_objects":4369,"num_object_clones":0,"num_object_copies":8738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4374,"num_write_kb":17467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":846},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":286580788,"num_objects":69983,"num_object_clones":0,"num_object_copies":139966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2938,"num_objects_unfound":0,"num_objects_dirty":69983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54038,"num_write_kb":216032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":562,"num_bytes_recovered":2220649,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":574795776,"data_stored":574708801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70045,"ondisk_log_size":70045,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051603,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559784,"kb_used_data":253600,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93812056,"statfs":{"total":96636764160,"available":96063545344,"internally_reserved":0,"allocated":259686400,"data_stored":259473244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":2.4489999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4910000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855366,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350076,"kb_used_data":39732,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94021764,"statfs":{"total":96636764160,"available":96278286336,"internally_reserved":0,"allocated":40685568,"data_stored":40392252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]}]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576276,"kb_used_data":265996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93795564,"statfs":{"total":96636764160,"available":96046657536,"internally_reserved":0,"allocated":272379904,"data_stored":272163126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,2,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":272347136,"data_stored":272343657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42102784,"data_stored":42022767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":260345856,"data_stored":260342377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:30.603 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:31.107 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 65 20 72482 72462 4.35367 2.16016 0.024671 0.0179336 2024-09-18T11:55:31.241 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:31.261 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:31.276 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-18T11:55:31.045429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":286580807,"num_objects":69985,"num_object_clones":0,"num_object_copies":139970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2938,"num_objects_unfound":0,"num_objects_dirty":69985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54040,"num_write_kb":216034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":2220668,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70048,"ondisk_log_size":70048,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1486136,"kb_used_data":559328,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281629384,"statfs":{"total":289910292480,"available":288388489216,"internally_reserved":0,"allocated":572751872,"data_stored":572028622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,10,2,5,15,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23891968,"num_objects":5833,"num_object_clones":0,"num_object_copies":11666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-242,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5833,"num_write_kb":23332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002857"},"pg_stats":[{"pgid":"2.f","version":"37'4437","reported_seq":4505,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.358096+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:26.358096+0000","last_peered":"2024-09-18T11:55:26.358096+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:26.358096+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18173952,"num_objects":4437,"num_object_clones":0,"num_object_copies":8874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4437,"num_write_kb":17748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":838},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4440","reported_seq":4596,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:27.492651+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:27.492651+0000","last_peered":"2024-09-18T11:55:27.492651+0000","last_clean":"2024-09-18T11:55:27.492651+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:27.492651+0000","last_undegraded":"2024-09-18T11:55:27.492651+0000","last_fullsized":"2024-09-18T11:55:27.492651+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18161664,"num_objects":4434,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1632,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4394","reported_seq":4571,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:27.475892+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:27.475892+0000","last_peered":"2024-09-18T11:55:27.475892+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:27.475892+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17981440,"num_objects":4390,"num_object_clones":0,"num_object_copies":8780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1252,"num_write_kb":5008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4358","reported_seq":4385,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:27.486995+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:27.486995+0000","last_peered":"2024-09-18T11:55:27.486995+0000","last_clean":"2024-09-18T11:55:27.486995+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:27.486995+0000","last_undegraded":"2024-09-18T11:55:27.486995+0000","last_fullsized":"2024-09-18T11:55:27.486995+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17850368,"num_objects":4358,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4358,"num_write_kb":17432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'4295","reported_seq":4484,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:26.335966+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:26.335966+0000","last_peered":"2024-09-18T11:55:26.335966+0000","last_clean":"2024-09-18T11:55:26.335966+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:26.335966+0000","last_undegraded":"2024-09-18T11:55:26.335966+0000","last_fullsized":"2024-09-18T11:55:26.335966+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":4295,"log_dups_size":0,"ondisk_log_size":4295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563648,"num_objects":4288,"num_object_clones":0,"num_object_copies":8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":5840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4304","reported_seq":4383,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.343714+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:26.343714+0000","last_peered":"2024-09-18T11:55:26.343714+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:26.343714+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17609938,"num_objects":4301,"num_object_clones":0,"num_object_copies":8602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":17199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":774},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'4348","reported_seq":4433,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.335805+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:26.335805+0000","last_peered":"2024-09-18T11:55:26.335805+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:26.335805+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4348,"log_dups_size":0,"ondisk_log_size":4348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17786066,"num_objects":4344,"num_object_clones":0,"num_object_copies":8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4346,"num_write_kb":17372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":785},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4462","reported_seq":4685,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:26.384506+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:26.384506+0000","last_peered":"2024-09-18T11:55:26.384506+0000","last_clean":"2024-09-18T11:55:26.384506+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:26.384506+0000","last_undegraded":"2024-09-18T11:55:26.384506+0000","last_fullsized":"2024-09-18T11:55:26.384506+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18268160,"num_objects":4460,"num_object_clones":0,"num_object_copies":8920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":779,"num_write_kb":3116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4323","reported_seq":5555,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:27.508559+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:27.508559+0000","last_peered":"2024-09-18T11:55:27.508559+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:27.508559+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17679570,"num_objects":4318,"num_object_clones":0,"num_object_copies":8636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2938,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":17268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":542,"num_bytes_recovered":2211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2938},{"shards":"2","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'4334","reported_seq":4410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.394335+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:26.394335+0000","last_peered":"2024-09-18T11:55:26.394335+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:26.394335+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4334,"log_dups_size":0,"ondisk_log_size":4334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17720530,"num_objects":4328,"num_object_clones":0,"num_object_copies":8656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4332,"num_write_kb":17308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":796},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4403","reported_seq":4457,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:27.510167+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:27.510167+0000","last_peered":"2024-09-18T11:55:27.510167+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:27.510167+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4403,"log_dups_size":0,"ondisk_log_size":4403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18015442,"num_objects":4400,"num_object_clones":0,"num_object_copies":8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4401,"num_write_kb":17595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":850}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4458","reported_seq":4665,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:27.437607+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:27.437607+0000","last_peered":"2024-09-18T11:55:27.437607+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:27.437607+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4458,"log_dups_size":0,"ondisk_log_size":4458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18235392,"num_objects":4452,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":938,"num_write_kb":3752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":859}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4366","reported_seq":4446,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.355516+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:26.355516+0000","last_peered":"2024-09-18T11:55:26.355516+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:26.355516+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17875561,"num_objects":4365,"num_object_clones":0,"num_object_copies":8730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4365,"num_write_kb":17457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":830},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'4369","reported_seq":4451,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.405171+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:26.405171+0000","last_peered":"2024-09-18T11:55:26.405171+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:26.405171+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17864507,"num_objects":4364,"num_object_clones":0,"num_object_copies":8728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4366,"num_write_kb":17448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":842},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4377","reported_seq":4456,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.319560+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:26.319560+0000","last_peered":"2024-09-18T11:55:26.319560+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:26.319560+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17913042,"num_objects":4375,"num_object_clones":0,"num_object_copies":8750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4375,"num_write_kb":17494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":782},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'4377","reported_seq":4460,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.351018+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:26.351018+0000","last_peered":"2024-09-18T11:55:26.351018+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:26.351018+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17881508,"num_objects":4369,"num_object_clones":0,"num_object_copies":8738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4374,"num_write_kb":17467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":846},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":286580788,"num_objects":69983,"num_object_clones":0,"num_object_copies":139966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2938,"num_objects_unfound":0,"num_objects_dirty":69983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54038,"num_write_kb":216032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":562,"num_bytes_recovered":2220649,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":574795776,"data_stored":574708801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70045,"ondisk_log_size":70045,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051603,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559784,"kb_used_data":253600,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93812056,"statfs":{"total":96636764160,"available":96063545344,"internally_reserved":0,"allocated":259686400,"data_stored":259473244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":2.4489999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4910000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855366,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350076,"kb_used_data":39732,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94021764,"statfs":{"total":96636764160,"available":96278286336,"internally_reserved":0,"allocated":40685568,"data_stored":40392252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]}]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576276,"kb_used_data":265996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93795564,"statfs":{"total":96636764160,"available":96046657536,"internally_reserved":0,"allocated":272379904,"data_stored":272163126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,2,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":272347136,"data_stored":272343657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42102784,"data_stored":42022767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":260345856,"data_stored":260342377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:31.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:55:31.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'4437", 'reported_seq': 4505, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:26.358096+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:55:26.358096+0000', 'last_peered': '2024-09-18T11:55:26.358096+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:55:26.358096+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4437, 'log_dups_size': 0, 'ondisk_log_size': 4437, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18173952, 'num_objects': 4437, 'num_object_clones': 0, 'num_object_copies': 8874, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4437, 'num_write_kb': 17748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 838}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:31.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:55:31.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'4394", 'reported_seq': 4571, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:27.475892+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:55:27.475892+0000', 'last_peered': '2024-09-18T11:55:27.475892+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:55:27.475892+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 4394, 'log_dups_size': 0, 'ondisk_log_size': 4394, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17981440, 'num_objects': 4390, 'num_object_clones': 0, 'num_object_copies': 8780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1252, 'num_write_kb': 5008, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 854}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:31.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:55:31.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'4304", 'reported_seq': 4383, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:26.343714+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:55:26.343714+0000', 'last_peered': '2024-09-18T11:55:26.343714+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:55:26.343714+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4304, 'log_dups_size': 0, 'ondisk_log_size': 4304, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17609938, 'num_objects': 4301, 'num_object_clones': 0, 'num_object_copies': 8602, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4302, 'num_write_kb': 17199, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 774}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:31.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:55:31.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'4348", 'reported_seq': 4433, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:26.335805+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:55:26.335805+0000', 'last_peered': '2024-09-18T11:55:26.335805+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:55:26.335805+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4348, 'log_dups_size': 0, 'ondisk_log_size': 4348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17786066, 'num_objects': 4344, 'num_object_clones': 0, 'num_object_copies': 8688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4344, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4346, 'num_write_kb': 17372, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 785}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:31.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:55:31.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'4323", 'reported_seq': 5555, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:27.508559+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:55:27.508559+0000', 'last_peered': '2024-09-18T11:55:27.508559+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:55:27.508559+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 4323, 'log_dups_size': 0, 'ondisk_log_size': 4323, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17679570, 'num_objects': 4318, 'num_object_clones': 0, 'num_object_copies': 8636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 2938, 'num_objects_unfound': 0, 'num_objects_dirty': 4318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4321, 'num_write_kb': 17268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 542, 'num_bytes_recovered': 2211840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2938}, {'shards': '2', 'objects': 1379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:31.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:55:31.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'4334", 'reported_seq': 4410, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:26.394335+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:55:26.394335+0000', 'last_peered': '2024-09-18T11:55:26.394335+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:55:26.394335+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4334, 'log_dups_size': 0, 'ondisk_log_size': 4334, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17720530, 'num_objects': 4328, 'num_object_clones': 0, 'num_object_copies': 8656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4332, 'num_write_kb': 17308, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 796}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:31.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:55:31.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'4403", 'reported_seq': 4457, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:27.510167+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:55:27.510167+0000', 'last_peered': '2024-09-18T11:55:27.510167+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:55:27.510167+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4403, 'log_dups_size': 0, 'ondisk_log_size': 4403, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18015442, 'num_objects': 4400, 'num_object_clones': 0, 'num_object_copies': 8800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4401, 'num_write_kb': 17595, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 850}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:31.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:55:31.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'4458", 'reported_seq': 4665, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:27.437607+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:55:27.437607+0000', 'last_peered': '2024-09-18T11:55:27.437607+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:55:27.437607+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 4458, 'log_dups_size': 0, 'ondisk_log_size': 4458, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18235392, 'num_objects': 4452, 'num_object_clones': 0, 'num_object_copies': 8904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 938, 'num_write_kb': 3752, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 859}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:31.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:55:31.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'4366", 'reported_seq': 4446, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:26.355516+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:55:26.355516+0000', 'last_peered': '2024-09-18T11:55:26.355516+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:55:26.355516+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4366, 'log_dups_size': 0, 'ondisk_log_size': 4366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17875561, 'num_objects': 4365, 'num_object_clones': 0, 'num_object_copies': 8730, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4365, 'num_write_kb': 17457, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 830}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:31.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:55:31.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'4369", 'reported_seq': 4451, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:26.405171+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:55:26.405171+0000', 'last_peered': '2024-09-18T11:55:26.405171+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:55:26.405171+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4369, 'log_dups_size': 0, 'ondisk_log_size': 4369, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17864507, 'num_objects': 4364, 'num_object_clones': 0, 'num_object_copies': 8728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4364, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4366, 'num_write_kb': 17448, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 842}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:31.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:55:31.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'4377", 'reported_seq': 4456, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:26.319560+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:55:26.319560+0000', 'last_peered': '2024-09-18T11:55:26.319560+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:55:26.319560+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4377, 'log_dups_size': 0, 'ondisk_log_size': 4377, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17913042, 'num_objects': 4375, 'num_object_clones': 0, 'num_object_copies': 8750, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4375, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4375, 'num_write_kb': 17494, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 782}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:31.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:55:31.280 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'4377", 'reported_seq': 4460, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:26.351018+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:55:26.351018+0000', 'last_peered': '2024-09-18T11:55:26.351018+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:55:26.351018+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4377, 'log_dups_size': 0, 'ondisk_log_size': 4377, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17881508, 'num_objects': 4369, 'num_object_clones': 0, 'num_object_copies': 8738, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4374, 'num_write_kb': 17467, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 846}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:31.280 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:32.107 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 66 20 73036 73016 4.3205 2.16406 0.0208028 0.0180734 2024-09-18T11:55:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:32.150 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/139970 objects degraded (27.927%), 12 pgs degraded","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":132,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":69985,"data_bytes":286580807,"bytes_used":1521803264,"bytes_avail":288388489216,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":139970,"degraded_ratio":0.27927413017075087,"misplaced_objects":2938,"misplaced_total":139970,"misplaced_ratio":0.020990212188326071,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":82583,"recovering_keys_per_sec":0,"num_objects_recovered":242,"num_bytes_recovered":991232,"num_keys_recovered":0,"write_bytes_sec":1990523,"read_op_per_sec":0,"write_op_per_sec":485},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:55:19.042648+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:55:32.176 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:32.176 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:32.903 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:32.920 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-18T11:55:31.045429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":286580807,"num_objects":69985,"num_object_clones":0,"num_object_copies":139970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2938,"num_objects_unfound":0,"num_objects_dirty":69985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54040,"num_write_kb":216034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":2220668,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70048,"ondisk_log_size":70048,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1486136,"kb_used_data":559328,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281629384,"statfs":{"total":289910292480,"available":288388489216,"internally_reserved":0,"allocated":572751872,"data_stored":572028622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,10,2,5,15,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23891968,"num_objects":5833,"num_object_clones":0,"num_object_copies":11666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-242,"num_objects_unfound":0,"num_objects_dirty":5833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5833,"num_write_kb":23332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002857"},"pg_stats":[{"pgid":"2.f","version":"37'4437","reported_seq":4505,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.358096+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:26.358096+0000","last_peered":"2024-09-18T11:55:26.358096+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:26.358096+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18173952,"num_objects":4437,"num_object_clones":0,"num_object_copies":8874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4437,"num_write_kb":17748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":838},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4440","reported_seq":4596,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:27.492651+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:27.492651+0000","last_peered":"2024-09-18T11:55:27.492651+0000","last_clean":"2024-09-18T11:55:27.492651+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:27.492651+0000","last_undegraded":"2024-09-18T11:55:27.492651+0000","last_fullsized":"2024-09-18T11:55:27.492651+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18161664,"num_objects":4434,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1632,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4394","reported_seq":4571,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:27.475892+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:27.475892+0000","last_peered":"2024-09-18T11:55:27.475892+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:27.475892+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17981440,"num_objects":4390,"num_object_clones":0,"num_object_copies":8780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1252,"num_write_kb":5008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4358","reported_seq":4385,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:27.486995+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:27.486995+0000","last_peered":"2024-09-18T11:55:27.486995+0000","last_clean":"2024-09-18T11:55:27.486995+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:27.486995+0000","last_undegraded":"2024-09-18T11:55:27.486995+0000","last_fullsized":"2024-09-18T11:55:27.486995+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17850368,"num_objects":4358,"num_object_clones":0,"num_object_copies":8716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4358,"num_write_kb":17432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'4295","reported_seq":4484,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:26.335966+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:26.335966+0000","last_peered":"2024-09-18T11:55:26.335966+0000","last_clean":"2024-09-18T11:55:26.335966+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:26.335966+0000","last_undegraded":"2024-09-18T11:55:26.335966+0000","last_fullsized":"2024-09-18T11:55:26.335966+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":4295,"log_dups_size":0,"ondisk_log_size":4295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563648,"num_objects":4288,"num_object_clones":0,"num_object_copies":8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":5840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4304","reported_seq":4383,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.343714+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:26.343714+0000","last_peered":"2024-09-18T11:55:26.343714+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:26.343714+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17609938,"num_objects":4301,"num_object_clones":0,"num_object_copies":8602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":17199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":774},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'4348","reported_seq":4433,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.335805+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:26.335805+0000","last_peered":"2024-09-18T11:55:26.335805+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:26.335805+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4348,"log_dups_size":0,"ondisk_log_size":4348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17786066,"num_objects":4344,"num_object_clones":0,"num_object_copies":8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4346,"num_write_kb":17372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":785},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4462","reported_seq":4685,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:26.384506+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:26.384506+0000","last_peered":"2024-09-18T11:55:26.384506+0000","last_clean":"2024-09-18T11:55:26.384506+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:26.384506+0000","last_undegraded":"2024-09-18T11:55:26.384506+0000","last_fullsized":"2024-09-18T11:55:26.384506+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18268160,"num_objects":4460,"num_object_clones":0,"num_object_copies":8920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":779,"num_write_kb":3116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4323","reported_seq":5555,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:27.508559+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:27.508559+0000","last_peered":"2024-09-18T11:55:27.508559+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:27.508559+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17679570,"num_objects":4318,"num_object_clones":0,"num_object_copies":8636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2938,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":17268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":542,"num_bytes_recovered":2211840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2938},{"shards":"2","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'4334","reported_seq":4410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.394335+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:26.394335+0000","last_peered":"2024-09-18T11:55:26.394335+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:26.394335+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4334,"log_dups_size":0,"ondisk_log_size":4334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17720530,"num_objects":4328,"num_object_clones":0,"num_object_copies":8656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4332,"num_write_kb":17308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":796},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4403","reported_seq":4457,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:27.510167+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:27.510167+0000","last_peered":"2024-09-18T11:55:27.510167+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:27.510167+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4403,"log_dups_size":0,"ondisk_log_size":4403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18015442,"num_objects":4400,"num_object_clones":0,"num_object_copies":8800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4401,"num_write_kb":17595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":850}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4458","reported_seq":4665,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:27.437607+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:27.437607+0000","last_peered":"2024-09-18T11:55:27.437607+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:27.437607+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4458,"log_dups_size":0,"ondisk_log_size":4458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18235392,"num_objects":4452,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":938,"num_write_kb":3752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":859}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4366","reported_seq":4446,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.355516+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:26.355516+0000","last_peered":"2024-09-18T11:55:26.355516+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:26.355516+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17875561,"num_objects":4365,"num_object_clones":0,"num_object_copies":8730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4365,"num_write_kb":17457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":830},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'4369","reported_seq":4451,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.405171+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:26.405171+0000","last_peered":"2024-09-18T11:55:26.405171+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:26.405171+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17864507,"num_objects":4364,"num_object_clones":0,"num_object_copies":8728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4366,"num_write_kb":17448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":842},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4377","reported_seq":4456,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.319560+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:26.319560+0000","last_peered":"2024-09-18T11:55:26.319560+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:26.319560+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17913042,"num_objects":4375,"num_object_clones":0,"num_object_copies":8750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4375,"num_write_kb":17494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":782},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'4377","reported_seq":4460,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:26.351018+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:26.351018+0000","last_peered":"2024-09-18T11:55:26.351018+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:26.351018+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17881508,"num_objects":4369,"num_object_clones":0,"num_object_copies":8738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4374,"num_write_kb":17467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":846},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":286580788,"num_objects":69983,"num_object_clones":0,"num_object_copies":139966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2938,"num_objects_unfound":0,"num_objects_dirty":69983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54038,"num_write_kb":216032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":562,"num_bytes_recovered":2220649,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":574795776,"data_stored":574708801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70045,"ondisk_log_size":70045,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051603,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559784,"kb_used_data":253600,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93812056,"statfs":{"total":96636764160,"available":96063545344,"internally_reserved":0,"allocated":259686400,"data_stored":259473244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":2.4489999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4910000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855366,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350076,"kb_used_data":39732,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94021764,"statfs":{"total":96636764160,"available":96278286336,"internally_reserved":0,"allocated":40685568,"data_stored":40392252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]}]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576276,"kb_used_data":265996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93795564,"statfs":{"total":96636764160,"available":96046657536,"internally_reserved":0,"allocated":272379904,"data_stored":272163126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,2,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":272347136,"data_stored":272343657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42102784,"data_stored":42022767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":260345856,"data_stored":260342377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:33.107 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 67 20 73572 73552 4.28727 2.09375 0.0713498 0.0182119 2024-09-18T11:55:34.108 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 68 20 74169 74149 4.25842 2.33203 0.00812896 0.0183352 2024-09-18T11:55:35.109 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 69 20 74754 74734 4.22982 2.28516 0.052803 0.0184612 2024-09-18T11:55:35.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:36.110 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 70 20 75330 75310 4.20144 2.25 0.0574311 0.0185856 2024-09-18T11:55:36.577 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:36.590 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:36.598 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-18T11:55:35.046299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298500167,"num_objects":72895,"num_object_clones":0,"num_object_copies":145790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2823,"num_objects_unfound":0,"num_objects_dirty":72895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56950,"num_write_kb":227674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":678,"num_bytes_recovered":2691708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72958,"ondisk_log_size":72958,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1493352,"kb_used_data":583184,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281622168,"statfs":{"total":289910292480,"available":288381100032,"internally_reserved":0,"allocated":597180416,"data_stored":596457166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,9,10,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23683072,"num_objects":5782,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-240,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5782,"num_write_kb":23128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002864"},"pg_stats":[{"pgid":"2.f","version":"37'4613","reported_seq":4682,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:31.401202+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:31.401202+0000","last_peered":"2024-09-18T11:55:31.401202+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:31.401202+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18894848,"num_objects":4613,"num_object_clones":0,"num_object_copies":9226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4613,"num_write_kb":18452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1014},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4611","reported_seq":4767,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:32.492955+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:32.492955+0000","last_peered":"2024-09-18T11:55:32.492955+0000","last_clean":"2024-09-18T11:55:32.492955+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:32.492955+0000","last_undegraded":"2024-09-18T11:55:32.492955+0000","last_fullsized":"2024-09-18T11:55:32.492955+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4611,"log_dups_size":0,"ondisk_log_size":4611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18862080,"num_objects":4605,"num_object_clones":0,"num_object_copies":9210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1803,"num_write_kb":7212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4589","reported_seq":4772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:32.442725+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:32.442725+0000","last_peered":"2024-09-18T11:55:32.442725+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:32.442725+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18780160,"num_objects":4585,"num_object_clones":0,"num_object_copies":9170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1447,"num_write_kb":5788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1049}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4536","reported_seq":4563,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:32.505951+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:32.505951+0000","last_peered":"2024-09-18T11:55:32.505951+0000","last_clean":"2024-09-18T11:55:32.505951+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:32.505951+0000","last_undegraded":"2024-09-18T11:55:32.505951+0000","last_fullsized":"2024-09-18T11:55:32.505951+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4536,"log_dups_size":0,"ondisk_log_size":4536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18579456,"num_objects":4536,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4536,"num_write_kb":18144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'4486","reported_seq":4675,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:31.415911+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:31.415911+0000","last_peered":"2024-09-18T11:55:31.415911+0000","last_clean":"2024-09-18T11:55:31.415911+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:31.415911+0000","last_undegraded":"2024-09-18T11:55:31.415911+0000","last_fullsized":"2024-09-18T11:55:31.415911+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":4486,"log_dups_size":0,"ondisk_log_size":4486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18345984,"num_objects":4479,"num_object_clones":0,"num_object_copies":8958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1651,"num_write_kb":6604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4476","reported_seq":4555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:31.408406+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:31.408406+0000","last_peered":"2024-09-18T11:55:31.408406+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:31.408406+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4476,"log_dups_size":0,"ondisk_log_size":4476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18314450,"num_objects":4473,"num_object_clones":0,"num_object_copies":8946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4474,"num_write_kb":17887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":946},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'4530","reported_seq":4613,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:31.277398+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:31.277398+0000","last_peered":"2024-09-18T11:55:31.277398+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:31.277398+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18531538,"num_objects":4526,"num_object_clones":0,"num_object_copies":9052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4528,"num_write_kb":18100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":967},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4671","reported_seq":4894,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:31.383228+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:31.383228+0000","last_peered":"2024-09-18T11:55:31.383228+0000","last_clean":"2024-09-18T11:55:31.383228+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:31.383228+0000","last_undegraded":"2024-09-18T11:55:31.383228+0000","last_fullsized":"2024-09-18T11:55:31.383228+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4671,"log_dups_size":0,"ondisk_log_size":4671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19124224,"num_objects":4669,"num_object_clones":0,"num_object_copies":9338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":988,"num_write_kb":3952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4501","reported_seq":5962,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:32.525647+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:32.525647+0000","last_peered":"2024-09-18T11:55:32.525647+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:32.525647+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18408658,"num_objects":4496,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2823,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4499,"num_write_kb":17980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":2682880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2823},{"shards":"2","objects":1672}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'4514","reported_seq":4590,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:31.389502+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:31.389502+0000","last_peered":"2024-09-18T11:55:31.389502+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:31.389502+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18457810,"num_objects":4508,"num_object_clones":0,"num_object_copies":9016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4512,"num_write_kb":18028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":976},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4579","reported_seq":4633,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:32.519509+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:32.519509+0000","last_peered":"2024-09-18T11:55:32.519509+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:32.519509+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18736338,"num_objects":4576,"num_object_clones":0,"num_object_copies":9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":18299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1026}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4619","reported_seq":4826,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:32.512196+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:32.512196+0000","last_peered":"2024-09-18T11:55:32.512196+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:32.512196+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18894848,"num_objects":4613,"num_object_clones":0,"num_object_copies":9226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":4396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1020}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4547","reported_seq":4627,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:31.340000+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:31.340000+0000","last_peered":"2024-09-18T11:55:31.340000+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:31.340000+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18616937,"num_objects":4546,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4546,"num_write_kb":18181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1011},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'4585","reported_seq":4667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:31.322306+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:31.322306+0000","last_peered":"2024-09-18T11:55:31.322306+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:31.322306+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4585,"log_dups_size":0,"ondisk_log_size":4585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18749243,"num_objects":4580,"num_object_clones":0,"num_object_copies":9160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4582,"num_write_kb":18312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1058},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4541","reported_seq":4620,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:31.359189+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:31.359189+0000","last_peered":"2024-09-18T11:55:31.359189+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:31.359189+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4541,"log_dups_size":0,"ondisk_log_size":4541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18584786,"num_objects":4539,"num_object_clones":0,"num_object_copies":9078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4539,"num_write_kb":18150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":946},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'4557","reported_seq":4640,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:31.337075+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:31.337075+0000","last_peered":"2024-09-18T11:55:31.337075+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:31.337075+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4557,"log_dups_size":0,"ondisk_log_size":4557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18618788,"num_objects":4549,"num_object_clones":0,"num_object_copies":9098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4554,"num_write_kb":18187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1026},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":298500148,"num_objects":72893,"num_object_clones":0,"num_object_copies":145786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2823,"num_objects_unfound":0,"num_objects_dirty":72893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56948,"num_write_kb":227672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":677,"num_bytes_recovered":2691689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":599138304,"data_stored":599051329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72955,"ondisk_log_size":72955,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051604,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569856,"kb_used_data":259512,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93801984,"statfs":{"total":96636764160,"available":96053231616,"internally_reserved":0,"allocated":265740288,"data_stored":265527132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.31},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":1.1819999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855367,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":351024,"kb_used_data":49000,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94020816,"statfs":{"total":96636764160,"available":96277315584,"internally_reserved":0,"allocated":50176000,"data_stored":49882684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1719999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572472,"kb_used_data":274672,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93799368,"statfs":{"total":96636764160,"available":96050552832,"internally_reserved":0,"allocated":281264128,"data_stored":281047350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.216},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.32600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":281358336,"data_stored":281354857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51466240,"data_stored":51386223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266313728,"data_stored":266310249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:36.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:37.110 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 71 20 75879 75859 4.17248 2.14453 0.0244086 0.0187138 2024-09-18T11:55:37.217 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:37.225 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:37.258 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-18T11:55:37.046694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":305963079,"num_objects":74717,"num_object_clones":0,"num_object_copies":149434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2823,"num_objects_unfound":0,"num_objects_dirty":74717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58772,"num_write_kb":234962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":678,"num_bytes_recovered":2691708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74780,"ondisk_log_size":74780,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1527828,"kb_used_data":601020,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281587692,"statfs":{"total":289910292480,"available":288345796608,"internally_reserved":0,"allocated":615444480,"data_stored":614721230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,7,11,8,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31145984,"num_objects":7604,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-240,"num_objects_unfound":0,"num_objects_dirty":7604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":30416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002777"},"pg_stats":[{"pgid":"2.f","version":"37'4796","reported_seq":4864,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.394895+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:36.394895+0000","last_peered":"2024-09-18T11:55:36.394895+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:36.394895+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19644416,"num_objects":4796,"num_object_clones":0,"num_object_copies":9592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4796,"num_write_kb":19184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1197},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4611","reported_seq":4767,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:32.492955+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:32.492955+0000","last_peered":"2024-09-18T11:55:32.492955+0000","last_clean":"2024-09-18T11:55:32.492955+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:32.492955+0000","last_undegraded":"2024-09-18T11:55:32.492955+0000","last_fullsized":"2024-09-18T11:55:32.492955+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4611,"log_dups_size":0,"ondisk_log_size":4611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18862080,"num_objects":4605,"num_object_clones":0,"num_object_copies":9210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1803,"num_write_kb":7212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4589","reported_seq":4772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:32.442725+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:32.442725+0000","last_peered":"2024-09-18T11:55:32.442725+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:32.442725+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18780160,"num_objects":4585,"num_object_clones":0,"num_object_copies":9170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1447,"num_write_kb":5788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1049}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4536","reported_seq":4563,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:32.505951+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:32.505951+0000","last_peered":"2024-09-18T11:55:32.505951+0000","last_clean":"2024-09-18T11:55:32.505951+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:32.505951+0000","last_undegraded":"2024-09-18T11:55:32.505951+0000","last_fullsized":"2024-09-18T11:55:32.505951+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4536,"log_dups_size":0,"ondisk_log_size":4536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18579456,"num_objects":4536,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4536,"num_write_kb":18144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'4663","reported_seq":4850,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:36.413023+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:36.413023+0000","last_peered":"2024-09-18T11:55:36.413023+0000","last_clean":"2024-09-18T11:55:36.413023+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:36.413023+0000","last_undegraded":"2024-09-18T11:55:36.413023+0000","last_fullsized":"2024-09-18T11:55:36.413023+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":4663,"log_dups_size":0,"ondisk_log_size":4663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19070976,"num_objects":4656,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1828,"num_write_kb":7312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4668","reported_seq":4747,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.371677+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:36.371677+0000","last_peered":"2024-09-18T11:55:36.371677+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:36.371677+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19100882,"num_objects":4665,"num_object_clones":0,"num_object_copies":9330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4666,"num_write_kb":18655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1138},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'4710","reported_seq":4790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.348785+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:36.348785+0000","last_peered":"2024-09-18T11:55:36.348785+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:36.348785+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19268818,"num_objects":4706,"num_object_clones":0,"num_object_copies":9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4708,"num_write_kb":18820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1147},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4855","reported_seq":5078,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:36.333307+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:36.333307+0000","last_peered":"2024-09-18T11:55:36.333307+0000","last_clean":"2024-09-18T11:55:36.333307+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:36.333307+0000","last_undegraded":"2024-09-18T11:55:36.333307+0000","last_fullsized":"2024-09-18T11:55:36.333307+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19877888,"num_objects":4853,"num_object_clones":0,"num_object_copies":9706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1172,"num_write_kb":4688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4501","reported_seq":5962,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:32.525647+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:32.525647+0000","last_peered":"2024-09-18T11:55:32.525647+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:32.525647+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18408658,"num_objects":4496,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2823,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4499,"num_write_kb":17980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":2682880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2823},{"shards":"2","objects":1672}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'4694","reported_seq":4770,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.335771+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:36.335771+0000","last_peered":"2024-09-18T11:55:36.335771+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:36.335771+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4694,"log_dups_size":0,"ondisk_log_size":4694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19195090,"num_objects":4688,"num_object_clones":0,"num_object_copies":9376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4692,"num_write_kb":18748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1156},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4579","reported_seq":4633,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:32.519509+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:32.519509+0000","last_peered":"2024-09-18T11:55:32.519509+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:32.519509+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18736338,"num_objects":4576,"num_object_clones":0,"num_object_copies":9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":18299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1026}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4619","reported_seq":4826,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:32.512196+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:32.512196+0000","last_peered":"2024-09-18T11:55:32.512196+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:32.512196+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18894848,"num_objects":4613,"num_object_clones":0,"num_object_copies":9226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":4396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1020}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4710","reported_seq":4790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.360831+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:36.360831+0000","last_peered":"2024-09-18T11:55:36.360831+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:36.360831+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19284585,"num_objects":4709,"num_object_clones":0,"num_object_copies":9418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":18833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1174},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'4771","reported_seq":4853,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.403348+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:36.403348+0000","last_peered":"2024-09-18T11:55:36.403348+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:36.403348+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4771,"log_dups_size":0,"ondisk_log_size":4771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19511099,"num_objects":4766,"num_object_clones":0,"num_object_copies":9532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4768,"num_write_kb":19056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1244},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4723","reported_seq":4802,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.393947+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:36.393947+0000","last_peered":"2024-09-18T11:55:36.393947+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:36.393947+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4723,"log_dups_size":0,"ondisk_log_size":4723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330258,"num_objects":4721,"num_object_clones":0,"num_object_copies":9442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4721,"num_write_kb":18878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1128},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'4752","reported_seq":4835,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.380278+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:36.380278+0000","last_peered":"2024-09-18T11:55:36.380278+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:36.380278+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4752,"log_dups_size":0,"ondisk_log_size":4752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19417508,"num_objects":4744,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":18967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1221},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":305963060,"num_objects":74715,"num_object_clones":0,"num_object_copies":149430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2823,"num_objects_unfound":0,"num_objects_dirty":74715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58770,"num_write_kb":234960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":677,"num_bytes_recovered":2691689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":617345024,"data_stored":617258049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74777,"ondisk_log_size":74777,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051604,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569856,"kb_used_data":259512,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93801984,"statfs":{"total":96636764160,"available":96053231616,"internally_reserved":0,"allocated":265740288,"data_stored":265527132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.31},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":1.1819999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855368,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368540,"kb_used_data":58196,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94003300,"statfs":{"total":96636764160,"available":96259379200,"internally_reserved":0,"allocated":59592704,"data_stored":59299388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0219999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589432,"kb_used_data":283312,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93782408,"statfs":{"total":96636764160,"available":96033185792,"internally_reserved":0,"allocated":290111488,"data_stored":289894710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,5,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.39500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":290193408,"data_stored":290189929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60837888,"data_stored":60757871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266313728,"data_stored":266310249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:37.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:55:37.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'4796", 'reported_seq': 4864, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:36.394895+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:55:36.394895+0000', 'last_peered': '2024-09-18T11:55:36.394895+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:55:36.394895+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4796, 'log_dups_size': 0, 'ondisk_log_size': 4796, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19644416, 'num_objects': 4796, 'num_object_clones': 0, 'num_object_copies': 9592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4796, 'num_write_kb': 19184, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1197}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:37.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:55:37.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'4589", 'reported_seq': 4772, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:32.442725+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:55:32.442725+0000', 'last_peered': '2024-09-18T11:55:32.442725+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:55:32.442725+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 4589, 'log_dups_size': 0, 'ondisk_log_size': 4589, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18780160, 'num_objects': 4585, 'num_object_clones': 0, 'num_object_copies': 9170, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1447, 'num_write_kb': 5788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 1049}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:37.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:55:37.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'4668", 'reported_seq': 4747, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:36.371677+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:55:36.371677+0000', 'last_peered': '2024-09-18T11:55:36.371677+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:55:36.371677+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4668, 'log_dups_size': 0, 'ondisk_log_size': 4668, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19100882, 'num_objects': 4665, 'num_object_clones': 0, 'num_object_copies': 9330, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4666, 'num_write_kb': 18655, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1138}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:37.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:55:37.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'4710", 'reported_seq': 4790, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:36.348785+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:55:36.348785+0000', 'last_peered': '2024-09-18T11:55:36.348785+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:55:36.348785+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4710, 'log_dups_size': 0, 'ondisk_log_size': 4710, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19268818, 'num_objects': 4706, 'num_object_clones': 0, 'num_object_copies': 9412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4708, 'num_write_kb': 18820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1147}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:37.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:55:37.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'4501", 'reported_seq': 5962, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:32.525647+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:55:32.525647+0000', 'last_peered': '2024-09-18T11:55:32.525647+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:55:32.525647+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 4501, 'log_dups_size': 0, 'ondisk_log_size': 4501, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18408658, 'num_objects': 4496, 'num_object_clones': 0, 'num_object_copies': 8992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 2823, 'num_objects_unfound': 0, 'num_objects_dirty': 4496, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4499, 'num_write_kb': 17980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 657, 'num_bytes_recovered': 2682880, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2823}, {'shards': '2', 'objects': 1672}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:37.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:55:37.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'4694", 'reported_seq': 4770, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:36.335771+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:55:36.335771+0000', 'last_peered': '2024-09-18T11:55:36.335771+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:55:36.335771+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4694, 'log_dups_size': 0, 'ondisk_log_size': 4694, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19195090, 'num_objects': 4688, 'num_object_clones': 0, 'num_object_copies': 9376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4692, 'num_write_kb': 18748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1156}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:37.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:55:37.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'4579", 'reported_seq': 4633, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:32.519509+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:55:32.519509+0000', 'last_peered': '2024-09-18T11:55:32.519509+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:55:32.519509+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4579, 'log_dups_size': 0, 'ondisk_log_size': 4579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18736338, 'num_objects': 4576, 'num_object_clones': 0, 'num_object_copies': 9152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4577, 'num_write_kb': 18299, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 1026}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:37.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:55:37.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'4619", 'reported_seq': 4826, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:32.512196+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:55:32.512196+0000', 'last_peered': '2024-09-18T11:55:32.512196+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:55:32.512196+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 4619, 'log_dups_size': 0, 'ondisk_log_size': 4619, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18894848, 'num_objects': 4613, 'num_object_clones': 0, 'num_object_copies': 9226, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1099, 'num_write_kb': 4396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 1020}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:37.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:55:37.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'4710", 'reported_seq': 4790, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:36.360831+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:55:36.360831+0000', 'last_peered': '2024-09-18T11:55:36.360831+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:55:36.360831+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4710, 'log_dups_size': 0, 'ondisk_log_size': 4710, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19284585, 'num_objects': 4709, 'num_object_clones': 0, 'num_object_copies': 9418, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4709, 'num_write_kb': 18833, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1174}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:37.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:55:37.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'4771", 'reported_seq': 4853, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:36.403348+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:55:36.403348+0000', 'last_peered': '2024-09-18T11:55:36.403348+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:55:36.403348+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4771, 'log_dups_size': 0, 'ondisk_log_size': 4771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19511099, 'num_objects': 4766, 'num_object_clones': 0, 'num_object_copies': 9532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4768, 'num_write_kb': 19056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1244}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:37.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:55:37.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'4723", 'reported_seq': 4802, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:36.393947+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:55:36.393947+0000', 'last_peered': '2024-09-18T11:55:36.393947+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:55:36.393947+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4723, 'log_dups_size': 0, 'ondisk_log_size': 4723, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19330258, 'num_objects': 4721, 'num_object_clones': 0, 'num_object_copies': 9442, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4721, 'num_write_kb': 18878, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1128}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:37.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:55:37.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'4752", 'reported_seq': 4835, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:36.380278+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:55:36.380278+0000', 'last_peered': '2024-09-18T11:55:36.380278+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:55:36.380278+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4752, 'log_dups_size': 0, 'ondisk_log_size': 4752, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19417508, 'num_objects': 4744, 'num_object_clones': 0, 'num_object_copies': 9488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4749, 'num_write_kb': 18967, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1221}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:37.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:38.030 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:38.030 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/149434 objects degraded (26.159%), 12 pgs degraded","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":138,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":74717,"data_bytes":305963079,"bytes_used":1564495872,"bytes_avail":288345796608,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":149434,"degraded_ratio":0.26158705515478403,"misplaced_objects":2823,"misplaced_total":149434,"misplaced_ratio":0.018891283108261842,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":81901,"recovering_keys_per_sec":0,"num_objects_recovered":240,"num_bytes_recovered":983040,"num_keys_recovered":0,"write_bytes_sec":2594898,"read_op_per_sec":0,"write_op_per_sec":633},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:55:19.042648+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:55:38.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:38.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:38.110 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 72 20 76422 76402 4.14399 2.12109 0.0071951 0.018843 2024-09-18T11:55:38.774 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:38.790 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:38.813 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-18T11:55:37.046694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":305963079,"num_objects":74717,"num_object_clones":0,"num_object_copies":149434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2823,"num_objects_unfound":0,"num_objects_dirty":74717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58772,"num_write_kb":234962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":678,"num_bytes_recovered":2691708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74780,"ondisk_log_size":74780,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1527828,"kb_used_data":601020,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281587692,"statfs":{"total":289910292480,"available":288345796608,"internally_reserved":0,"allocated":615444480,"data_stored":614721230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,7,11,8,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31145984,"num_objects":7604,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-240,"num_objects_unfound":0,"num_objects_dirty":7604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":30416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002777"},"pg_stats":[{"pgid":"2.f","version":"37'4796","reported_seq":4864,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.394895+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:36.394895+0000","last_peered":"2024-09-18T11:55:36.394895+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:36.394895+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19644416,"num_objects":4796,"num_object_clones":0,"num_object_copies":9592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4796,"num_write_kb":19184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1197},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4611","reported_seq":4767,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:32.492955+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:32.492955+0000","last_peered":"2024-09-18T11:55:32.492955+0000","last_clean":"2024-09-18T11:55:32.492955+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:32.492955+0000","last_undegraded":"2024-09-18T11:55:32.492955+0000","last_fullsized":"2024-09-18T11:55:32.492955+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4611,"log_dups_size":0,"ondisk_log_size":4611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18862080,"num_objects":4605,"num_object_clones":0,"num_object_copies":9210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1803,"num_write_kb":7212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4589","reported_seq":4772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:32.442725+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:32.442725+0000","last_peered":"2024-09-18T11:55:32.442725+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:32.442725+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18780160,"num_objects":4585,"num_object_clones":0,"num_object_copies":9170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1447,"num_write_kb":5788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1049}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4536","reported_seq":4563,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:32.505951+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:32.505951+0000","last_peered":"2024-09-18T11:55:32.505951+0000","last_clean":"2024-09-18T11:55:32.505951+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:32.505951+0000","last_undegraded":"2024-09-18T11:55:32.505951+0000","last_fullsized":"2024-09-18T11:55:32.505951+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4536,"log_dups_size":0,"ondisk_log_size":4536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18579456,"num_objects":4536,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4536,"num_write_kb":18144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'4663","reported_seq":4850,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:36.413023+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:36.413023+0000","last_peered":"2024-09-18T11:55:36.413023+0000","last_clean":"2024-09-18T11:55:36.413023+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:36.413023+0000","last_undegraded":"2024-09-18T11:55:36.413023+0000","last_fullsized":"2024-09-18T11:55:36.413023+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":4663,"log_dups_size":0,"ondisk_log_size":4663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19070976,"num_objects":4656,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1828,"num_write_kb":7312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4668","reported_seq":4747,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.371677+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:36.371677+0000","last_peered":"2024-09-18T11:55:36.371677+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:36.371677+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19100882,"num_objects":4665,"num_object_clones":0,"num_object_copies":9330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4666,"num_write_kb":18655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1138},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'4710","reported_seq":4790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.348785+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:36.348785+0000","last_peered":"2024-09-18T11:55:36.348785+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:36.348785+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19268818,"num_objects":4706,"num_object_clones":0,"num_object_copies":9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4708,"num_write_kb":18820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1147},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4855","reported_seq":5078,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:36.333307+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:36.333307+0000","last_peered":"2024-09-18T11:55:36.333307+0000","last_clean":"2024-09-18T11:55:36.333307+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:36.333307+0000","last_undegraded":"2024-09-18T11:55:36.333307+0000","last_fullsized":"2024-09-18T11:55:36.333307+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19877888,"num_objects":4853,"num_object_clones":0,"num_object_copies":9706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1172,"num_write_kb":4688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4501","reported_seq":5962,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:32.525647+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:32.525647+0000","last_peered":"2024-09-18T11:55:32.525647+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:32.525647+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18408658,"num_objects":4496,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2823,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4499,"num_write_kb":17980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":2682880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2823},{"shards":"2","objects":1672}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'4694","reported_seq":4770,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.335771+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:36.335771+0000","last_peered":"2024-09-18T11:55:36.335771+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:36.335771+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4694,"log_dups_size":0,"ondisk_log_size":4694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19195090,"num_objects":4688,"num_object_clones":0,"num_object_copies":9376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4692,"num_write_kb":18748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1156},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4579","reported_seq":4633,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:32.519509+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:32.519509+0000","last_peered":"2024-09-18T11:55:32.519509+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:32.519509+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18736338,"num_objects":4576,"num_object_clones":0,"num_object_copies":9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":18299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1026}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4619","reported_seq":4826,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:32.512196+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:32.512196+0000","last_peered":"2024-09-18T11:55:32.512196+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:32.512196+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18894848,"num_objects":4613,"num_object_clones":0,"num_object_copies":9226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":4396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1020}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4710","reported_seq":4790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.360831+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:36.360831+0000","last_peered":"2024-09-18T11:55:36.360831+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:36.360831+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19284585,"num_objects":4709,"num_object_clones":0,"num_object_copies":9418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":18833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1174},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'4771","reported_seq":4853,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.403348+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:36.403348+0000","last_peered":"2024-09-18T11:55:36.403348+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:36.403348+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4771,"log_dups_size":0,"ondisk_log_size":4771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19511099,"num_objects":4766,"num_object_clones":0,"num_object_copies":9532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4768,"num_write_kb":19056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1244},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4723","reported_seq":4802,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.393947+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:36.393947+0000","last_peered":"2024-09-18T11:55:36.393947+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:36.393947+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4723,"log_dups_size":0,"ondisk_log_size":4723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330258,"num_objects":4721,"num_object_clones":0,"num_object_copies":9442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4721,"num_write_kb":18878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1128},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'4752","reported_seq":4835,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.380278+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:36.380278+0000","last_peered":"2024-09-18T11:55:36.380278+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:36.380278+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4752,"log_dups_size":0,"ondisk_log_size":4752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19417508,"num_objects":4744,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":18967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1221},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":305963060,"num_objects":74715,"num_object_clones":0,"num_object_copies":149430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2823,"num_objects_unfound":0,"num_objects_dirty":74715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58770,"num_write_kb":234960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":677,"num_bytes_recovered":2691689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":617345024,"data_stored":617258049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74777,"ondisk_log_size":74777,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051604,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569856,"kb_used_data":259512,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93801984,"statfs":{"total":96636764160,"available":96053231616,"internally_reserved":0,"allocated":265740288,"data_stored":265527132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.31},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":1.1819999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855368,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368540,"kb_used_data":58196,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94003300,"statfs":{"total":96636764160,"available":96259379200,"internally_reserved":0,"allocated":59592704,"data_stored":59299388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0219999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589432,"kb_used_data":283312,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93782408,"statfs":{"total":96636764160,"available":96033185792,"internally_reserved":0,"allocated":290111488,"data_stored":289894710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,5,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.39500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":290193408,"data_stored":290189929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60837888,"data_stored":60757871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266313728,"data_stored":266310249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:39.110 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 73 20 76978 76958 4.11697 2.17188 0.0277464 0.0189685 2024-09-18T11:55:40.110 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 74 20 77568 77548 4.09249 2.30469 0.043985 0.0190821 2024-09-18T11:55:41.110 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 75 20 78144 78124 4.06792 2.25 0.0208924 0.0191963 2024-09-18T11:55:41.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:42.111 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 76 20 78734 78714 4.04472 2.30469 0.0490971 0.019307 2024-09-18T11:55:42.393 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:42.416 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:42.434 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-18T11:55:41.047552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":310317127,"num_objects":75780,"num_object_clones":0,"num_object_copies":151560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2699,"num_objects_unfound":0,"num_objects_dirty":75780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59835,"num_write_kb":239214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":802,"num_bytes_recovered":3199612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75843,"ondisk_log_size":75843,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1521104,"kb_used_data":606776,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281594416,"statfs":{"total":289910292480,"available":288352681984,"internally_reserved":0,"allocated":621338624,"data_stored":620615374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,14,8,10,8,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23736320,"num_objects":5795,"num_object_clones":0,"num_object_copies":11590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-239,"num_objects_unfound":0,"num_objects_dirty":5795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5795,"num_write_kb":23180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":239,"num_bytes_recovered":978944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002549"},"pg_stats":[{"pgid":"2.f","version":"37'4796","reported_seq":4864,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.394895+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:36.394895+0000","last_peered":"2024-09-18T11:55:36.394895+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:36.394895+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19644416,"num_objects":4796,"num_object_clones":0,"num_object_copies":9592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4796,"num_write_kb":19184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1197},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4774","reported_seq":4930,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:37.471484+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:37.471484+0000","last_peered":"2024-09-18T11:55:37.471484+0000","last_clean":"2024-09-18T11:55:37.471484+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:37.471484+0000","last_undegraded":"2024-09-18T11:55:37.471484+0000","last_fullsized":"2024-09-18T11:55:37.471484+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4774,"log_dups_size":0,"ondisk_log_size":4774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19529728,"num_objects":4768,"num_object_clones":0,"num_object_copies":9536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1966,"num_write_kb":7864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4767","reported_seq":4950,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:37.558433+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:37.558433+0000","last_peered":"2024-09-18T11:55:37.558433+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:37.558433+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4767,"log_dups_size":0,"ondisk_log_size":4767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19509248,"num_objects":4763,"num_object_clones":0,"num_object_copies":9526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1625,"num_write_kb":6500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1227}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4727","reported_seq":4754,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:37.501358+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:37.501358+0000","last_peered":"2024-09-18T11:55:37.501358+0000","last_clean":"2024-09-18T11:55:37.501358+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:37.501358+0000","last_undegraded":"2024-09-18T11:55:37.501358+0000","last_fullsized":"2024-09-18T11:55:37.501358+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4727,"log_dups_size":0,"ondisk_log_size":4727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19361792,"num_objects":4727,"num_object_clones":0,"num_object_copies":9454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":18908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'4663","reported_seq":4850,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:36.413023+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:36.413023+0000","last_peered":"2024-09-18T11:55:36.413023+0000","last_clean":"2024-09-18T11:55:36.413023+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:36.413023+0000","last_undegraded":"2024-09-18T11:55:36.413023+0000","last_fullsized":"2024-09-18T11:55:36.413023+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":4663,"log_dups_size":0,"ondisk_log_size":4663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19070976,"num_objects":4656,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1828,"num_write_kb":7312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4668","reported_seq":4747,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.371677+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:36.371677+0000","last_peered":"2024-09-18T11:55:36.371677+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:36.371677+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19100882,"num_objects":4665,"num_object_clones":0,"num_object_copies":9330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4666,"num_write_kb":18655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1138},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'4710","reported_seq":4790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.348785+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:36.348785+0000","last_peered":"2024-09-18T11:55:36.348785+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:36.348785+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19268818,"num_objects":4706,"num_object_clones":0,"num_object_copies":9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4708,"num_write_kb":18820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1147},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'4855","reported_seq":5078,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:36.333307+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:36.333307+0000","last_peered":"2024-09-18T11:55:36.333307+0000","last_clean":"2024-09-18T11:55:36.333307+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:36.333307+0000","last_undegraded":"2024-09-18T11:55:36.333307+0000","last_fullsized":"2024-09-18T11:55:36.333307+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19877888,"num_objects":4853,"num_object_clones":0,"num_object_copies":9706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1172,"num_write_kb":4688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4680","reported_seq":6386,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:37.548349+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:37.548349+0000","last_peered":"2024-09-18T11:55:37.548349+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:37.548349+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4680,"log_dups_size":0,"ondisk_log_size":4680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19141842,"num_objects":4675,"num_object_clones":0,"num_object_copies":9350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2699,"num_objects_unfound":0,"num_objects_dirty":4675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4678,"num_write_kb":18696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":781,"num_bytes_recovered":3190784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2699},{"shards":"2","objects":1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'4694","reported_seq":4770,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.335771+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:36.335771+0000","last_peered":"2024-09-18T11:55:36.335771+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:36.335771+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4694,"log_dups_size":0,"ondisk_log_size":4694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19195090,"num_objects":4688,"num_object_clones":0,"num_object_copies":9376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4692,"num_write_kb":18748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1156},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4771","reported_seq":4825,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:37.399918+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:37.399918+0000","last_peered":"2024-09-18T11:55:37.399918+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:37.399918+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4771,"log_dups_size":0,"ondisk_log_size":4771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19522770,"num_objects":4768,"num_object_clones":0,"num_object_copies":9536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4769,"num_write_kb":19067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4779","reported_seq":4985,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:37.518754+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:37.518754+0000","last_peered":"2024-09-18T11:55:37.518754+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:37.518754+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19550208,"num_objects":4773,"num_object_clones":0,"num_object_copies":9546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1259,"num_write_kb":5036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1180}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4710","reported_seq":4790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.360831+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:36.360831+0000","last_peered":"2024-09-18T11:55:36.360831+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:36.360831+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19284585,"num_objects":4709,"num_object_clones":0,"num_object_copies":9418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":18833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1174},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'4771","reported_seq":4853,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.403348+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:36.403348+0000","last_peered":"2024-09-18T11:55:36.403348+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:36.403348+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4771,"log_dups_size":0,"ondisk_log_size":4771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19511099,"num_objects":4766,"num_object_clones":0,"num_object_copies":9532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4768,"num_write_kb":19056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1244},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4723","reported_seq":4802,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.393947+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:36.393947+0000","last_peered":"2024-09-18T11:55:36.393947+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:36.393947+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4723,"log_dups_size":0,"ondisk_log_size":4723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19330258,"num_objects":4721,"num_object_clones":0,"num_object_copies":9442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4721,"num_write_kb":18878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1128},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'4752","reported_seq":4835,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:36.380278+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:36.380278+0000","last_peered":"2024-09-18T11:55:36.380278+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:36.380278+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4752,"log_dups_size":0,"ondisk_log_size":4752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19417508,"num_objects":4744,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":18967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1221},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":310317108,"num_objects":75778,"num_object_clones":0,"num_object_copies":151556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2699,"num_objects_unfound":0,"num_objects_dirty":75778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59833,"num_write_kb":239212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":3199593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":623198208,"data_stored":623111233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75840,"ondisk_log_size":75840,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051605,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563132,"kb_used_data":265268,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93808708,"statfs":{"total":96636764160,"available":96060116992,"internally_reserved":0,"allocated":271634432,"data_stored":271421276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,10,1,1,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":7.3639999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4080000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855368,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368540,"kb_used_data":58196,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94003300,"statfs":{"total":96636764160,"available":96259379200,"internally_reserved":0,"allocated":59592704,"data_stored":59299388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0219999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589432,"kb_used_data":283312,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93782408,"statfs":{"total":96636764160,"available":96033185792,"internally_reserved":0,"allocated":290111488,"data_stored":289894710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,5,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.39500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":290193408,"data_stored":290189929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60837888,"data_stored":60757871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":272166912,"data_stored":272163433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:42.435 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:43.111 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 77 20 79272 79252 4.01949 2.10156 0.0197214 0.0194234 2024-09-18T11:55:43.172 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:43.180 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:43.195 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-18T11:55:43.047968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":321892423,"num_objects":78606,"num_object_clones":0,"num_object_copies":157212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":78606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62661,"num_write_kb":250518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":920,"num_bytes_recovered":3682940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78669,"ondisk_log_size":78669,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1540156,"kb_used_data":629988,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281575364,"statfs":{"total":289910292480,"available":288333172736,"internally_reserved":0,"allocated":645107712,"data_stored":644384462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,11,11,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35311616,"num_objects":8621,"num_object_clones":0,"num_object_copies":17242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-357,"num_objects_unfound":0,"num_objects_dirty":8621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8621,"num_write_kb":34484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":357,"num_bytes_recovered":1462272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002539"},"pg_stats":[{"pgid":"2.f","version":"37'4981","reported_seq":5050,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.386667+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:41.386667+0000","last_peered":"2024-09-18T11:55:41.386667+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:41.386667+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20402176,"num_objects":4981,"num_object_clones":0,"num_object_copies":9962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4981,"num_write_kb":19924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1382},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4962","reported_seq":5118,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.554279+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:42.554279+0000","last_peered":"2024-09-18T11:55:42.554279+0000","last_clean":"2024-09-18T11:55:42.554279+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:42.554279+0000","last_undegraded":"2024-09-18T11:55:42.554279+0000","last_fullsized":"2024-09-18T11:55:42.554279+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20299776,"num_objects":4956,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2154,"num_write_kb":8616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4943","reported_seq":5126,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.559729+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:42.559729+0000","last_peered":"2024-09-18T11:55:42.559729+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:42.559729+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4943,"log_dups_size":0,"ondisk_log_size":4943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20230144,"num_objects":4939,"num_object_clones":0,"num_object_copies":9878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1801,"num_write_kb":7204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4913","reported_seq":4940,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.563729+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:42.563729+0000","last_peered":"2024-09-18T11:55:42.563729+0000","last_clean":"2024-09-18T11:55:42.563729+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:42.563729+0000","last_undegraded":"2024-09-18T11:55:42.563729+0000","last_fullsized":"2024-09-18T11:55:42.563729+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20123648,"num_objects":4913,"num_object_clones":0,"num_object_copies":9826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":19652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'4854","reported_seq":5042,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:41.407745+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:41.407745+0000","last_peered":"2024-09-18T11:55:41.407745+0000","last_clean":"2024-09-18T11:55:41.407745+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:41.407745+0000","last_undegraded":"2024-09-18T11:55:41.407745+0000","last_fullsized":"2024-09-18T11:55:41.407745+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19853312,"num_objects":4847,"num_object_clones":0,"num_object_copies":9694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2019,"num_write_kb":8076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4820","reported_seq":4899,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.404016+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:41.404016+0000","last_peered":"2024-09-18T11:55:41.404016+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:41.404016+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19723474,"num_objects":4817,"num_object_clones":0,"num_object_copies":9634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":19263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1290},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'4887","reported_seq":4972,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.410247+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:41.410247+0000","last_peered":"2024-09-18T11:55:41.410247+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:41.410247+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19993810,"num_objects":4883,"num_object_clones":0,"num_object_copies":9766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4885,"num_write_kb":19528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1324},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5024","reported_seq":5247,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:41.371820+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:41.371820+0000","last_peered":"2024-09-18T11:55:41.371820+0000","last_clean":"2024-09-18T11:55:41.371820+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:41.371820+0000","last_undegraded":"2024-09-18T11:55:41.371820+0000","last_fullsized":"2024-09-18T11:55:41.371820+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5024,"log_dups_size":0,"ondisk_log_size":5024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20570112,"num_objects":5022,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1341,"num_write_kb":5364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4864","reported_seq":6810,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.579274+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:42.579274+0000","last_peered":"2024-09-18T11:55:42.579274+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:42.579274+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19895506,"num_objects":4859,"num_object_clones":0,"num_object_copies":9718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":19432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":899,"num_bytes_recovered":3674112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2581},{"shards":"2","objects":2277}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'4867","reported_seq":4943,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.277424+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:41.277424+0000","last_peered":"2024-09-18T11:55:41.277424+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:41.277424+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19903698,"num_objects":4861,"num_object_clones":0,"num_object_copies":9722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4865,"num_write_kb":19440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1329},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4966","reported_seq":5020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.483909+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:42.483909+0000","last_peered":"2024-09-18T11:55:42.483909+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:42.483909+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4966,"log_dups_size":0,"ondisk_log_size":4966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20321490,"num_objects":4963,"num_object_clones":0,"num_object_copies":9926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4964,"num_write_kb":19847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4947","reported_seq":5154,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.480396+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:42.480396+0000","last_peered":"2024-09-18T11:55:42.480396+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:42.480396+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4947,"log_dups_size":0,"ondisk_log_size":4947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20238336,"num_objects":4941,"num_object_clones":0,"num_object_copies":9882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":5708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1348}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4888","reported_seq":4968,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.391521+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:41.391521+0000","last_peered":"2024-09-18T11:55:41.391521+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:41.391521+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20013673,"num_objects":4887,"num_object_clones":0,"num_object_copies":9774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4887,"num_write_kb":19545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1352},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'4941","reported_seq":5023,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.433505+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:41.433505+0000","last_peered":"2024-09-18T11:55:41.433505+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:41.433505+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20207419,"num_objects":4936,"num_object_clones":0,"num_object_copies":9872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4938,"num_write_kb":19736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1414},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4883","reported_seq":4959,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.415227+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:41.415227+0000","last_peered":"2024-09-18T11:55:41.415227+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:41.415227+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4883,"log_dups_size":0,"ondisk_log_size":4883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19985618,"num_objects":4881,"num_object_clones":0,"num_object_copies":9762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4881,"num_write_kb":19518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1288},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'4926","reported_seq":5009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.247620+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:41.247620+0000","last_peered":"2024-09-18T11:55:41.247620+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:41.247620+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20130212,"num_objects":4918,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4923,"num_write_kb":19663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1395},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":321892404,"num_objects":78604,"num_object_clones":0,"num_object_copies":157208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":78604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62659,"num_write_kb":250516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":919,"num_bytes_recovered":3682921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":646983680,"data_stored":646896705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78666,"ondisk_log_size":78666,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051606,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573172,"kb_used_data":271148,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93798668,"statfs":{"total":96636764160,"available":96049836032,"internally_reserved":0,"allocated":277655552,"data_stored":277442396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":2.8980000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855369,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364960,"kb_used_data":67096,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94006880,"statfs":{"total":96636764160,"available":96263045120,"internally_reserved":0,"allocated":68706304,"data_stored":68412988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602024,"kb_used_data":291744,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93769816,"statfs":{"total":96636764160,"available":96020291584,"internally_reserved":0,"allocated":298745856,"data_stored":298529078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,7,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.34399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298827776,"data_stored":298824297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69992448,"data_stored":69912431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278163456,"data_stored":278159977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:43.196 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:55:43.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'4981", 'reported_seq': 5050, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:41.386667+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:55:41.386667+0000', 'last_peered': '2024-09-18T11:55:41.386667+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:55:41.386667+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4981, 'log_dups_size': 0, 'ondisk_log_size': 4981, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20402176, 'num_objects': 4981, 'num_object_clones': 0, 'num_object_copies': 9962, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4981, 'num_write_kb': 19924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1382}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:43.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:55:43.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'4943", 'reported_seq': 5126, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:42.559729+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:55:42.559729+0000', 'last_peered': '2024-09-18T11:55:42.559729+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:55:42.559729+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 4943, 'log_dups_size': 0, 'ondisk_log_size': 4943, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20230144, 'num_objects': 4939, 'num_object_clones': 0, 'num_object_copies': 9878, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1801, 'num_write_kb': 7204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 1403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:43.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:55:43.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'4820", 'reported_seq': 4899, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:41.404016+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:55:41.404016+0000', 'last_peered': '2024-09-18T11:55:41.404016+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:55:41.404016+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4820, 'log_dups_size': 0, 'ondisk_log_size': 4820, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19723474, 'num_objects': 4817, 'num_object_clones': 0, 'num_object_copies': 9634, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4818, 'num_write_kb': 19263, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1290}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:43.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:55:43.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'4887", 'reported_seq': 4972, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:41.410247+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:55:41.410247+0000', 'last_peered': '2024-09-18T11:55:41.410247+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:55:41.410247+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4887, 'log_dups_size': 0, 'ondisk_log_size': 4887, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19993810, 'num_objects': 4883, 'num_object_clones': 0, 'num_object_copies': 9766, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4885, 'num_write_kb': 19528, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1324}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:43.197 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:55:43.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'4864", 'reported_seq': 6810, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:42.579274+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:55:42.579274+0000', 'last_peered': '2024-09-18T11:55:42.579274+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:55:42.579274+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 4864, 'log_dups_size': 0, 'ondisk_log_size': 4864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19895506, 'num_objects': 4859, 'num_object_clones': 0, 'num_object_copies': 9718, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 2581, 'num_objects_unfound': 0, 'num_objects_dirty': 4859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4862, 'num_write_kb': 19432, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 899, 'num_bytes_recovered': 3674112, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2581}, {'shards': '2', 'objects': 2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:43.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:55:43.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'4867", 'reported_seq': 4943, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:41.277424+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:55:41.277424+0000', 'last_peered': '2024-09-18T11:55:41.277424+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:55:41.277424+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4867, 'log_dups_size': 0, 'ondisk_log_size': 4867, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19903698, 'num_objects': 4861, 'num_object_clones': 0, 'num_object_copies': 9722, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4865, 'num_write_kb': 19440, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1329}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:43.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:55:43.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'4966", 'reported_seq': 5020, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:42.483909+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:55:42.483909+0000', 'last_peered': '2024-09-18T11:55:42.483909+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:55:42.483909+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4966, 'log_dups_size': 0, 'ondisk_log_size': 4966, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20321490, 'num_objects': 4963, 'num_object_clones': 0, 'num_object_copies': 9926, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4963, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4964, 'num_write_kb': 19847, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 1413}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:43.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:55:43.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'4947", 'reported_seq': 5154, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:42.480396+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:55:42.480396+0000', 'last_peered': '2024-09-18T11:55:42.480396+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:55:42.480396+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 4947, 'log_dups_size': 0, 'ondisk_log_size': 4947, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20238336, 'num_objects': 4941, 'num_object_clones': 0, 'num_object_copies': 9882, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1427, 'num_write_kb': 5708, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 1348}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:43.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:55:43.198 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'4888", 'reported_seq': 4968, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:41.391521+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:55:41.391521+0000', 'last_peered': '2024-09-18T11:55:41.391521+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:55:41.391521+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4888, 'log_dups_size': 0, 'ondisk_log_size': 4888, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20013673, 'num_objects': 4887, 'num_object_clones': 0, 'num_object_copies': 9774, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4887, 'num_write_kb': 19545, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1352}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:43.199 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:55:43.199 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'4941", 'reported_seq': 5023, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:41.433505+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:55:41.433505+0000', 'last_peered': '2024-09-18T11:55:41.433505+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:55:41.433505+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4941, 'log_dups_size': 0, 'ondisk_log_size': 4941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20207419, 'num_objects': 4936, 'num_object_clones': 0, 'num_object_copies': 9872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4938, 'num_write_kb': 19736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1414}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:43.199 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:55:43.199 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'4883", 'reported_seq': 4959, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:41.415227+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:55:41.415227+0000', 'last_peered': '2024-09-18T11:55:41.415227+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:55:41.415227+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4883, 'log_dups_size': 0, 'ondisk_log_size': 4883, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19985618, 'num_objects': 4881, 'num_object_clones': 0, 'num_object_copies': 9762, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4881, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4881, 'num_write_kb': 19518, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1288}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:43.199 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:55:43.199 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'4926", 'reported_seq': 5009, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:41.247620+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:55:41.247620+0000', 'last_peered': '2024-09-18T11:55:41.247620+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:55:41.247620+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4926, 'log_dups_size': 0, 'ondisk_log_size': 4926, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20130212, 'num_objects': 4918, 'num_object_clones': 0, 'num_object_copies': 9836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4918, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4923, 'num_write_kb': 19663, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1395}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:43.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:44.114 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 78 20 79808 79788 3.9948 2.09375 0.026297 0.0195482 2024-09-18T11:55:44.114 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:44.114 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/157212 objects degraded (24.865%), 12 pgs degraded","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":144,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":78606,"data_bytes":321892423,"bytes_used":1577119744,"bytes_avail":288333172736,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":157212,"degraded_ratio":0.24864514159224488,"misplaced_objects":2581,"misplaced_total":157212,"misplaced_ratio":0.016417321832938963,"recovering_objects_per_sec":29,"recovering_bytes_per_sec":121830,"recovering_keys_per_sec":0,"num_objects_recovered":357,"num_bytes_recovered":1462272,"num_keys_recovered":0,"write_bytes_sec":2942012,"read_op_per_sec":0,"write_op_per_sec":718},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:55:19.042648+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:55:44.140 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:44.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:44.638 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:44.654 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-18T11:55:43.047968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":321892423,"num_objects":78606,"num_object_clones":0,"num_object_copies":157212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":78606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62661,"num_write_kb":250518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":920,"num_bytes_recovered":3682940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78669,"ondisk_log_size":78669,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1540156,"kb_used_data":629988,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281575364,"statfs":{"total":289910292480,"available":288333172736,"internally_reserved":0,"allocated":645107712,"data_stored":644384462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,11,11,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35311616,"num_objects":8621,"num_object_clones":0,"num_object_copies":17242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-357,"num_objects_unfound":0,"num_objects_dirty":8621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8621,"num_write_kb":34484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":357,"num_bytes_recovered":1462272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002539"},"pg_stats":[{"pgid":"2.f","version":"37'4981","reported_seq":5050,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.386667+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:41.386667+0000","last_peered":"2024-09-18T11:55:41.386667+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:41.386667+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20402176,"num_objects":4981,"num_object_clones":0,"num_object_copies":9962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4981,"num_write_kb":19924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1382},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4962","reported_seq":5118,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.554279+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:42.554279+0000","last_peered":"2024-09-18T11:55:42.554279+0000","last_clean":"2024-09-18T11:55:42.554279+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:42.554279+0000","last_undegraded":"2024-09-18T11:55:42.554279+0000","last_fullsized":"2024-09-18T11:55:42.554279+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20299776,"num_objects":4956,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2154,"num_write_kb":8616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4943","reported_seq":5126,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.559729+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:42.559729+0000","last_peered":"2024-09-18T11:55:42.559729+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:42.559729+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4943,"log_dups_size":0,"ondisk_log_size":4943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20230144,"num_objects":4939,"num_object_clones":0,"num_object_copies":9878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1801,"num_write_kb":7204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4913","reported_seq":4940,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.563729+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:42.563729+0000","last_peered":"2024-09-18T11:55:42.563729+0000","last_clean":"2024-09-18T11:55:42.563729+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:42.563729+0000","last_undegraded":"2024-09-18T11:55:42.563729+0000","last_fullsized":"2024-09-18T11:55:42.563729+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20123648,"num_objects":4913,"num_object_clones":0,"num_object_copies":9826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":19652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'4854","reported_seq":5042,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:41.407745+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:41.407745+0000","last_peered":"2024-09-18T11:55:41.407745+0000","last_clean":"2024-09-18T11:55:41.407745+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:41.407745+0000","last_undegraded":"2024-09-18T11:55:41.407745+0000","last_fullsized":"2024-09-18T11:55:41.407745+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19853312,"num_objects":4847,"num_object_clones":0,"num_object_copies":9694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2019,"num_write_kb":8076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4820","reported_seq":4899,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.404016+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:41.404016+0000","last_peered":"2024-09-18T11:55:41.404016+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:41.404016+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19723474,"num_objects":4817,"num_object_clones":0,"num_object_copies":9634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":19263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1290},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'4887","reported_seq":4972,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.410247+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:41.410247+0000","last_peered":"2024-09-18T11:55:41.410247+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:41.410247+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19993810,"num_objects":4883,"num_object_clones":0,"num_object_copies":9766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4885,"num_write_kb":19528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1324},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5024","reported_seq":5247,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:41.371820+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:41.371820+0000","last_peered":"2024-09-18T11:55:41.371820+0000","last_clean":"2024-09-18T11:55:41.371820+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:41.371820+0000","last_undegraded":"2024-09-18T11:55:41.371820+0000","last_fullsized":"2024-09-18T11:55:41.371820+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5024,"log_dups_size":0,"ondisk_log_size":5024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20570112,"num_objects":5022,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1341,"num_write_kb":5364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4864","reported_seq":6810,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.579274+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:42.579274+0000","last_peered":"2024-09-18T11:55:42.579274+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:42.579274+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19895506,"num_objects":4859,"num_object_clones":0,"num_object_copies":9718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":19432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":899,"num_bytes_recovered":3674112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2581},{"shards":"2","objects":2277}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'4867","reported_seq":4943,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.277424+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:41.277424+0000","last_peered":"2024-09-18T11:55:41.277424+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:41.277424+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19903698,"num_objects":4861,"num_object_clones":0,"num_object_copies":9722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4865,"num_write_kb":19440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1329},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4966","reported_seq":5020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.483909+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:42.483909+0000","last_peered":"2024-09-18T11:55:42.483909+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:42.483909+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4966,"log_dups_size":0,"ondisk_log_size":4966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20321490,"num_objects":4963,"num_object_clones":0,"num_object_copies":9926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4964,"num_write_kb":19847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4947","reported_seq":5154,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.480396+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:42.480396+0000","last_peered":"2024-09-18T11:55:42.480396+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:42.480396+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4947,"log_dups_size":0,"ondisk_log_size":4947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20238336,"num_objects":4941,"num_object_clones":0,"num_object_copies":9882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":5708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1348}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'4888","reported_seq":4968,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.391521+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:41.391521+0000","last_peered":"2024-09-18T11:55:41.391521+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:41.391521+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20013673,"num_objects":4887,"num_object_clones":0,"num_object_copies":9774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4887,"num_write_kb":19545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1352},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'4941","reported_seq":5023,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.433505+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:41.433505+0000","last_peered":"2024-09-18T11:55:41.433505+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:41.433505+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20207419,"num_objects":4936,"num_object_clones":0,"num_object_copies":9872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4938,"num_write_kb":19736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1414},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'4883","reported_seq":4959,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.415227+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:41.415227+0000","last_peered":"2024-09-18T11:55:41.415227+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:41.415227+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4883,"log_dups_size":0,"ondisk_log_size":4883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19985618,"num_objects":4881,"num_object_clones":0,"num_object_copies":9762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4881,"num_write_kb":19518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1288},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'4926","reported_seq":5009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:41.247620+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:41.247620+0000","last_peered":"2024-09-18T11:55:41.247620+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:41.247620+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20130212,"num_objects":4918,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4923,"num_write_kb":19663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1395},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":321892404,"num_objects":78604,"num_object_clones":0,"num_object_copies":157208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":78604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62659,"num_write_kb":250516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":919,"num_bytes_recovered":3682921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":646983680,"data_stored":646896705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78666,"ondisk_log_size":78666,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051606,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573172,"kb_used_data":271148,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93798668,"statfs":{"total":96636764160,"available":96049836032,"internally_reserved":0,"allocated":277655552,"data_stored":277442396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":2.8980000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855369,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364960,"kb_used_data":67096,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94006880,"statfs":{"total":96636764160,"available":96263045120,"internally_reserved":0,"allocated":68706304,"data_stored":68412988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602024,"kb_used_data":291744,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93769816,"statfs":{"total":96636764160,"available":96020291584,"internally_reserved":0,"allocated":298745856,"data_stored":298529078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,7,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.34399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298827776,"data_stored":298824297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69992448,"data_stored":69912431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278163456,"data_stored":278159977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:45.111 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 79 20 80314 80294 3.96925 1.97656 0.0199312 0.0196748 2024-09-18T11:55:46.112 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:2024-09-18T11:55:46.111722+0000 min lat: 0.0021114 max lat: 2.57663 avg lat: 0.0197817 2024-09-18T11:55:46.112 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:55:46.113 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 80 20 80880 80860 3.94728 2.21094 0.0581927 0.0197817 2024-09-18T11:55:47.111 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 81 20 81455 81435 3.92627 2.24609 0.00897937 0.0198874 2024-09-18T11:55:47.656 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:48.115 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 82 20 82007 81987 3.90463 2.15625 0.0157293 0.0199974 2024-09-18T11:55:48.243 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:48.262 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:48.282 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-18T11:55:47.050612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328806471,"num_objects":80294,"num_object_clones":0,"num_object_copies":160588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":80294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64349,"num_write_kb":257270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":920,"num_bytes_recovered":3682940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80357,"ondisk_log_size":80357,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1557076,"kb_used_data":646908,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281558444,"statfs":{"total":289910292480,"available":288315846656,"internally_reserved":0,"allocated":662433792,"data_stored":661710542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,3,11,13,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30306304,"num_objects":7399,"num_object_clones":0,"num_object_copies":14798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-242,"num_objects_unfound":0,"num_objects_dirty":7399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7399,"num_write_kb":29596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004312"},"pg_stats":[{"pgid":"2.f","version":"37'5141","reported_seq":5210,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.357191+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:46.357191+0000","last_peered":"2024-09-18T11:55:46.357191+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:46.357191+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5141,"log_dups_size":0,"ondisk_log_size":5141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21057536,"num_objects":5141,"num_object_clones":0,"num_object_copies":10282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5141,"num_write_kb":20564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1542},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4962","reported_seq":5118,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.554279+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:42.554279+0000","last_peered":"2024-09-18T11:55:42.554279+0000","last_clean":"2024-09-18T11:55:42.554279+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:42.554279+0000","last_undegraded":"2024-09-18T11:55:42.554279+0000","last_fullsized":"2024-09-18T11:55:42.554279+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20299776,"num_objects":4956,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2154,"num_write_kb":8616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4943","reported_seq":5126,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.559729+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:42.559729+0000","last_peered":"2024-09-18T11:55:42.559729+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:42.559729+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4943,"log_dups_size":0,"ondisk_log_size":4943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20230144,"num_objects":4939,"num_object_clones":0,"num_object_copies":9878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1801,"num_write_kb":7204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4913","reported_seq":4940,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.563729+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:42.563729+0000","last_peered":"2024-09-18T11:55:42.563729+0000","last_clean":"2024-09-18T11:55:42.563729+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:42.563729+0000","last_undegraded":"2024-09-18T11:55:42.563729+0000","last_fullsized":"2024-09-18T11:55:42.563729+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20123648,"num_objects":4913,"num_object_clones":0,"num_object_copies":9826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":19652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5025","reported_seq":5214,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:46.295564+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:46.295564+0000","last_peered":"2024-09-18T11:55:46.295564+0000","last_clean":"2024-09-18T11:55:46.295564+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:46.295564+0000","last_undegraded":"2024-09-18T11:55:46.295564+0000","last_fullsized":"2024-09-18T11:55:46.295564+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20553728,"num_objects":5018,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2190,"num_write_kb":8760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4996","reported_seq":5075,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.448022+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:46.448022+0000","last_peered":"2024-09-18T11:55:46.448022+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:46.448022+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4996,"log_dups_size":0,"ondisk_log_size":4996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20444370,"num_objects":4993,"num_object_clones":0,"num_object_copies":9986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4994,"num_write_kb":19967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1466},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5046","reported_seq":5130,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.427301+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:46.427301+0000","last_peered":"2024-09-18T11:55:46.427301+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:46.427301+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20645074,"num_objects":5042,"num_object_clones":0,"num_object_copies":10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":20164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1483},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5187","reported_seq":5409,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:46.442810+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:46.442810+0000","last_peered":"2024-09-18T11:55:46.442810+0000","last_clean":"2024-09-18T11:55:46.442810+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:46.442810+0000","last_undegraded":"2024-09-18T11:55:46.442810+0000","last_fullsized":"2024-09-18T11:55:46.442810+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21237760,"num_objects":5185,"num_object_clones":0,"num_object_copies":10370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1504,"num_write_kb":6016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4864","reported_seq":6810,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.579274+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:42.579274+0000","last_peered":"2024-09-18T11:55:42.579274+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:42.579274+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19895506,"num_objects":4859,"num_object_clones":0,"num_object_copies":9718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":19432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":899,"num_bytes_recovered":3674112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2581},{"shards":"2","objects":2277}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5052","reported_seq":5128,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.464854+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:46.464854+0000","last_peered":"2024-09-18T11:55:46.464854+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:46.464854+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20661458,"num_objects":5046,"num_object_clones":0,"num_object_copies":10092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5050,"num_write_kb":20180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1514},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4966","reported_seq":5020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.483909+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:42.483909+0000","last_peered":"2024-09-18T11:55:42.483909+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:42.483909+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4966,"log_dups_size":0,"ondisk_log_size":4966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20321490,"num_objects":4963,"num_object_clones":0,"num_object_copies":9926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4964,"num_write_kb":19847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4947","reported_seq":5154,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.480396+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:42.480396+0000","last_peered":"2024-09-18T11:55:42.480396+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:42.480396+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4947,"log_dups_size":0,"ondisk_log_size":4947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20238336,"num_objects":4941,"num_object_clones":0,"num_object_copies":9882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":5708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1348}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5052","reported_seq":5131,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.417144+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:46.417144+0000","last_peered":"2024-09-18T11:55:46.417144+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:46.417144+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20685417,"num_objects":5051,"num_object_clones":0,"num_object_copies":10102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":20201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1516},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5114","reported_seq":5196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.421013+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:46.421013+0000","last_peered":"2024-09-18T11:55:46.421013+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:46.421013+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5114,"log_dups_size":0,"ondisk_log_size":5114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20916027,"num_objects":5109,"num_object_clones":0,"num_object_copies":10218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5111,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1587},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5046","reported_seq":5125,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.469592+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:46.469592+0000","last_peered":"2024-09-18T11:55:46.469592+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:46.469592+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20653266,"num_objects":5044,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":20170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1451},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5100","reported_seq":5183,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.468241+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:46.468241+0000","last_peered":"2024-09-18T11:55:46.468241+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:46.468241+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20842916,"num_objects":5092,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5097,"num_write_kb":20359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1569},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":328806452,"num_objects":80292,"num_object_clones":0,"num_object_copies":160584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":80292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64347,"num_write_kb":257268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":919,"num_bytes_recovered":3682921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":664244224,"data_stored":664157249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80354,"ondisk_log_size":80354,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051606,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573172,"kb_used_data":271148,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93798668,"statfs":{"total":96636764160,"available":96049836032,"internally_reserved":0,"allocated":277655552,"data_stored":277442396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":2.8980000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855370,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381920,"kb_used_data":75736,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93989920,"statfs":{"total":96636764160,"available":96245678080,"internally_reserved":0,"allocated":77553664,"data_stored":77260348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.214}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601984,"kb_used_data":300024,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93769856,"statfs":{"total":96636764160,"available":96020332544,"internally_reserved":0,"allocated":307224576,"data_stored":307007798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":307130368,"data_stored":307126889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78950400,"data_stored":78870383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278163456,"data_stored":278159977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:48.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:48.866 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:48.881 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:48.897 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-18T11:55:47.050612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328806471,"num_objects":80294,"num_object_clones":0,"num_object_copies":160588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":80294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64349,"num_write_kb":257270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":920,"num_bytes_recovered":3682940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80357,"ondisk_log_size":80357,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1557076,"kb_used_data":646908,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281558444,"statfs":{"total":289910292480,"available":288315846656,"internally_reserved":0,"allocated":662433792,"data_stored":661710542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,3,11,13,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30306304,"num_objects":7399,"num_object_clones":0,"num_object_copies":14798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-242,"num_objects_unfound":0,"num_objects_dirty":7399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7399,"num_write_kb":29596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004312"},"pg_stats":[{"pgid":"2.f","version":"37'5141","reported_seq":5210,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.357191+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:46.357191+0000","last_peered":"2024-09-18T11:55:46.357191+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:46.357191+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5141,"log_dups_size":0,"ondisk_log_size":5141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21057536,"num_objects":5141,"num_object_clones":0,"num_object_copies":10282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5141,"num_write_kb":20564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1542},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'4962","reported_seq":5118,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.554279+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:42.554279+0000","last_peered":"2024-09-18T11:55:42.554279+0000","last_clean":"2024-09-18T11:55:42.554279+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:42.554279+0000","last_undegraded":"2024-09-18T11:55:42.554279+0000","last_fullsized":"2024-09-18T11:55:42.554279+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20299776,"num_objects":4956,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2154,"num_write_kb":8616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'4943","reported_seq":5126,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.559729+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:42.559729+0000","last_peered":"2024-09-18T11:55:42.559729+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:42.559729+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":4943,"log_dups_size":0,"ondisk_log_size":4943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20230144,"num_objects":4939,"num_object_clones":0,"num_object_copies":9878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1801,"num_write_kb":7204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'4913","reported_seq":4940,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:42.563729+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:42.563729+0000","last_peered":"2024-09-18T11:55:42.563729+0000","last_clean":"2024-09-18T11:55:42.563729+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:42.563729+0000","last_undegraded":"2024-09-18T11:55:42.563729+0000","last_fullsized":"2024-09-18T11:55:42.563729+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20123648,"num_objects":4913,"num_object_clones":0,"num_object_copies":9826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":19652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5025","reported_seq":5214,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:46.295564+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:46.295564+0000","last_peered":"2024-09-18T11:55:46.295564+0000","last_clean":"2024-09-18T11:55:46.295564+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:46.295564+0000","last_undegraded":"2024-09-18T11:55:46.295564+0000","last_fullsized":"2024-09-18T11:55:46.295564+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20553728,"num_objects":5018,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2190,"num_write_kb":8760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4996","reported_seq":5075,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.448022+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:46.448022+0000","last_peered":"2024-09-18T11:55:46.448022+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:46.448022+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4996,"log_dups_size":0,"ondisk_log_size":4996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20444370,"num_objects":4993,"num_object_clones":0,"num_object_copies":9986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4994,"num_write_kb":19967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1466},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5046","reported_seq":5130,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.427301+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:46.427301+0000","last_peered":"2024-09-18T11:55:46.427301+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:46.427301+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20645074,"num_objects":5042,"num_object_clones":0,"num_object_copies":10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":20164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1483},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5187","reported_seq":5409,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:46.442810+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:46.442810+0000","last_peered":"2024-09-18T11:55:46.442810+0000","last_clean":"2024-09-18T11:55:46.442810+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:46.442810+0000","last_undegraded":"2024-09-18T11:55:46.442810+0000","last_fullsized":"2024-09-18T11:55:46.442810+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21237760,"num_objects":5185,"num_object_clones":0,"num_object_copies":10370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1504,"num_write_kb":6016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'4864","reported_seq":6810,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.579274+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:42.579274+0000","last_peered":"2024-09-18T11:55:42.579274+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:42.579274+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19895506,"num_objects":4859,"num_object_clones":0,"num_object_copies":9718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":19432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":899,"num_bytes_recovered":3674112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2581},{"shards":"2","objects":2277}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5052","reported_seq":5128,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.464854+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:46.464854+0000","last_peered":"2024-09-18T11:55:46.464854+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:46.464854+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20661458,"num_objects":5046,"num_object_clones":0,"num_object_copies":10092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5050,"num_write_kb":20180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1514},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'4966","reported_seq":5020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.483909+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:42.483909+0000","last_peered":"2024-09-18T11:55:42.483909+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:42.483909+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4966,"log_dups_size":0,"ondisk_log_size":4966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20321490,"num_objects":4963,"num_object_clones":0,"num_object_copies":9926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4964,"num_write_kb":19847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4947","reported_seq":5154,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:42.480396+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:42.480396+0000","last_peered":"2024-09-18T11:55:42.480396+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:42.480396+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":4947,"log_dups_size":0,"ondisk_log_size":4947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20238336,"num_objects":4941,"num_object_clones":0,"num_object_copies":9882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":5708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1348}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5052","reported_seq":5131,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.417144+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:46.417144+0000","last_peered":"2024-09-18T11:55:46.417144+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:46.417144+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20685417,"num_objects":5051,"num_object_clones":0,"num_object_copies":10102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":20201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1516},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5114","reported_seq":5196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.421013+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:46.421013+0000","last_peered":"2024-09-18T11:55:46.421013+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:46.421013+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5114,"log_dups_size":0,"ondisk_log_size":5114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20916027,"num_objects":5109,"num_object_clones":0,"num_object_copies":10218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5111,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1587},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5046","reported_seq":5125,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.469592+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:46.469592+0000","last_peered":"2024-09-18T11:55:46.469592+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:46.469592+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20653266,"num_objects":5044,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":20170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1451},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5100","reported_seq":5183,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.468241+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:46.468241+0000","last_peered":"2024-09-18T11:55:46.468241+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:46.468241+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20842916,"num_objects":5092,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5097,"num_write_kb":20359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1569},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":328806452,"num_objects":80292,"num_object_clones":0,"num_object_copies":160584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":80292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64347,"num_write_kb":257268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":919,"num_bytes_recovered":3682921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":664244224,"data_stored":664157249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80354,"ondisk_log_size":80354,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051606,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573172,"kb_used_data":271148,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93798668,"statfs":{"total":96636764160,"available":96049836032,"internally_reserved":0,"allocated":277655552,"data_stored":277442396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":2.8980000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.37}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855370,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381920,"kb_used_data":75736,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93989920,"statfs":{"total":96636764160,"available":96245678080,"internally_reserved":0,"allocated":77553664,"data_stored":77260348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.214}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601984,"kb_used_data":300024,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93769856,"statfs":{"total":96636764160,"available":96020332544,"internally_reserved":0,"allocated":307224576,"data_stored":307007798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":307130368,"data_stored":307126889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78950400,"data_stored":78870383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278163456,"data_stored":278159977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:48.898 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:55:48.898 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'5141", 'reported_seq': 5210, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:46.357191+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:55:46.357191+0000', 'last_peered': '2024-09-18T11:55:46.357191+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:55:46.357191+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5141, 'log_dups_size': 0, 'ondisk_log_size': 5141, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21057536, 'num_objects': 5141, 'num_object_clones': 0, 'num_object_copies': 10282, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5141, 'num_write_kb': 20564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1542}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:48.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:55:48.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'4943", 'reported_seq': 5126, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:42.559729+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:55:42.559729+0000', 'last_peered': '2024-09-18T11:55:42.559729+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:55:42.559729+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 4943, 'log_dups_size': 0, 'ondisk_log_size': 4943, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20230144, 'num_objects': 4939, 'num_object_clones': 0, 'num_object_copies': 9878, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1801, 'num_write_kb': 7204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 1403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:48.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:55:48.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'4996", 'reported_seq': 5075, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:46.448022+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:55:46.448022+0000', 'last_peered': '2024-09-18T11:55:46.448022+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:55:46.448022+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4996, 'log_dups_size': 0, 'ondisk_log_size': 4996, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20444370, 'num_objects': 4993, 'num_object_clones': 0, 'num_object_copies': 9986, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4993, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4994, 'num_write_kb': 19967, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1466}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:48.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:55:48.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'5046", 'reported_seq': 5130, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:46.427301+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:55:46.427301+0000', 'last_peered': '2024-09-18T11:55:46.427301+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:55:46.427301+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5046, 'log_dups_size': 0, 'ondisk_log_size': 5046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20645074, 'num_objects': 5042, 'num_object_clones': 0, 'num_object_copies': 10084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5042, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5044, 'num_write_kb': 20164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1483}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:48.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:55:48.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'4864", 'reported_seq': 6810, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:42.579274+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:55:42.579274+0000', 'last_peered': '2024-09-18T11:55:42.579274+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:55:42.579274+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 4864, 'log_dups_size': 0, 'ondisk_log_size': 4864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19895506, 'num_objects': 4859, 'num_object_clones': 0, 'num_object_copies': 9718, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 2581, 'num_objects_unfound': 0, 'num_objects_dirty': 4859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4862, 'num_write_kb': 19432, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 899, 'num_bytes_recovered': 3674112, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2581}, {'shards': '2', 'objects': 2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:48.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:55:48.900 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'5052", 'reported_seq': 5128, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:46.464854+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:55:46.464854+0000', 'last_peered': '2024-09-18T11:55:46.464854+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:55:46.464854+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5052, 'log_dups_size': 0, 'ondisk_log_size': 5052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20661458, 'num_objects': 5046, 'num_object_clones': 0, 'num_object_copies': 10092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5050, 'num_write_kb': 20180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1514}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:48.900 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:55:48.900 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'4966", 'reported_seq': 5020, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:42.483909+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:55:42.483909+0000', 'last_peered': '2024-09-18T11:55:42.483909+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:55:42.483909+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 4966, 'log_dups_size': 0, 'ondisk_log_size': 4966, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20321490, 'num_objects': 4963, 'num_object_clones': 0, 'num_object_copies': 9926, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4963, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4964, 'num_write_kb': 19847, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 1413}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:48.900 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:55:48.900 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'4947", 'reported_seq': 5154, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:42.480396+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:55:42.480396+0000', 'last_peered': '2024-09-18T11:55:42.480396+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:55:42.480396+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 4947, 'log_dups_size': 0, 'ondisk_log_size': 4947, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20238336, 'num_objects': 4941, 'num_object_clones': 0, 'num_object_copies': 9882, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1427, 'num_write_kb': 5708, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 1348}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:48.900 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:55:48.900 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'5052", 'reported_seq': 5131, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:46.417144+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:55:46.417144+0000', 'last_peered': '2024-09-18T11:55:46.417144+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:55:46.417144+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5052, 'log_dups_size': 0, 'ondisk_log_size': 5052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20685417, 'num_objects': 5051, 'num_object_clones': 0, 'num_object_copies': 10102, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5051, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5051, 'num_write_kb': 20201, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1516}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:48.900 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:55:48.900 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'5114", 'reported_seq': 5196, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:46.421013+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:55:46.421013+0000', 'last_peered': '2024-09-18T11:55:46.421013+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:55:46.421013+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5114, 'log_dups_size': 0, 'ondisk_log_size': 5114, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20916027, 'num_objects': 5109, 'num_object_clones': 0, 'num_object_copies': 10218, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5111, 'num_write_kb': 20428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1587}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:48.901 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:55:48.901 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'5046", 'reported_seq': 5125, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:46.469592+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:55:46.469592+0000', 'last_peered': '2024-09-18T11:55:46.469592+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:55:46.469592+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5046, 'log_dups_size': 0, 'ondisk_log_size': 5046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20653266, 'num_objects': 5044, 'num_object_clones': 0, 'num_object_copies': 10088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5044, 'num_write_kb': 20170, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1451}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:48.901 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:55:48.901 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'5100", 'reported_seq': 5183, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:46.468241+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:55:46.468241+0000', 'last_peered': '2024-09-18T11:55:46.468241+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:55:46.468241+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5100, 'log_dups_size': 0, 'ondisk_log_size': 5100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20842916, 'num_objects': 5092, 'num_object_clones': 0, 'num_object_copies': 10184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5092, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5097, 'num_write_kb': 20359, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1569}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:48.901 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:49.112 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 83 20 82523 82503 3.88188 2.01562 0.0237743 0.020116 2024-09-18T11:55:49.896 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:49.896 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/162668 objects degraded (24.031%), 12 pgs degraded","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":149,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":81334,"data_bytes":333066311,"bytes_used":1608450048,"bytes_avail":288301842432,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":162668,"degraded_ratio":0.24030540733272679,"misplaced_objects":2465,"misplaced_total":162668,"misplaced_ratio":0.015153564315046598,"recovering_objects_per_sec":29,"recovering_bytes_per_sec":122152,"recovering_keys_per_sec":0,"num_objects_recovered":358,"num_bytes_recovered":1466368,"num_keys_recovered":0,"write_bytes_sec":2257777,"read_op_per_sec":0,"write_op_per_sec":551},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:55:19.042648+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:55:49.937 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:49.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:50.112 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 84 20 83062 83042 3.86073 2.10547 0.0494119 0.0202256 2024-09-18T11:55:50.516 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:50.530 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:50.537 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-18T11:55:49.051082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":333066311,"num_objects":81334,"num_object_clones":0,"num_object_copies":162668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2465,"num_objects_unfound":0,"num_objects_dirty":81334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65389,"num_write_kb":261430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1036,"num_bytes_recovered":4158076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81397,"ondisk_log_size":81397,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1570752,"kb_used_data":652264,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281544768,"statfs":{"total":289910292480,"available":288301842432,"internally_reserved":0,"allocated":667918336,"data_stored":667195086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,2,9,8,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27103232,"num_objects":6617,"num_object_clones":0,"num_object_copies":13234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-358,"num_objects_unfound":0,"num_objects_dirty":6617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6617,"num_write_kb":26468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":358,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004387"},"pg_stats":[{"pgid":"2.f","version":"37'5141","reported_seq":5210,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.357191+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:46.357191+0000","last_peered":"2024-09-18T11:55:46.357191+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:46.357191+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5141,"log_dups_size":0,"ondisk_log_size":5141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21057536,"num_objects":5141,"num_object_clones":0,"num_object_copies":10282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5141,"num_write_kb":20564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1542},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5119","reported_seq":5275,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:47.532524+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:47.532524+0000","last_peered":"2024-09-18T11:55:47.532524+0000","last_clean":"2024-09-18T11:55:47.532524+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:47.532524+0000","last_undegraded":"2024-09-18T11:55:47.532524+0000","last_fullsized":"2024-09-18T11:55:47.532524+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5119,"log_dups_size":0,"ondisk_log_size":5119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20942848,"num_objects":5113,"num_object_clones":0,"num_object_copies":10226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2311,"num_write_kb":9244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5125","reported_seq":5306,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:47.569815+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:47.569815+0000","last_peered":"2024-09-18T11:55:47.569815+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:47.569815+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5125,"log_dups_size":0,"ondisk_log_size":5125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20975616,"num_objects":5121,"num_object_clones":0,"num_object_copies":10242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1983,"num_write_kb":7932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1585}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5079","reported_seq":5106,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:47.557324+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:47.557324+0000","last_peered":"2024-09-18T11:55:47.557324+0000","last_clean":"2024-09-18T11:55:47.557324+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:47.557324+0000","last_undegraded":"2024-09-18T11:55:47.557324+0000","last_fullsized":"2024-09-18T11:55:47.557324+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20803584,"num_objects":5079,"num_object_clones":0,"num_object_copies":10158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5079,"num_write_kb":20316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5025","reported_seq":5214,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:46.295564+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:46.295564+0000","last_peered":"2024-09-18T11:55:46.295564+0000","last_clean":"2024-09-18T11:55:46.295564+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:46.295564+0000","last_undegraded":"2024-09-18T11:55:46.295564+0000","last_fullsized":"2024-09-18T11:55:46.295564+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20553728,"num_objects":5018,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2190,"num_write_kb":8760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'4996","reported_seq":5075,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.448022+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:46.448022+0000","last_peered":"2024-09-18T11:55:46.448022+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:46.448022+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":4996,"log_dups_size":0,"ondisk_log_size":4996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20444370,"num_objects":4993,"num_object_clones":0,"num_object_copies":9986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4994,"num_write_kb":19967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1466},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5046","reported_seq":5130,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.427301+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:46.427301+0000","last_peered":"2024-09-18T11:55:46.427301+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:46.427301+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20645074,"num_objects":5042,"num_object_clones":0,"num_object_copies":10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":20164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1483},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5187","reported_seq":5409,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:46.442810+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:46.442810+0000","last_peered":"2024-09-18T11:55:46.442810+0000","last_clean":"2024-09-18T11:55:46.442810+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:46.442810+0000","last_undegraded":"2024-09-18T11:55:46.442810+0000","last_fullsized":"2024-09-18T11:55:46.442810+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21237760,"num_objects":5185,"num_object_clones":0,"num_object_copies":10370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1504,"num_write_kb":6016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5050","reported_seq":7227,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:47.548538+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:47.548538+0000","last_peered":"2024-09-18T11:55:47.548538+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:47.548538+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657362,"num_objects":5045,"num_object_clones":0,"num_object_copies":10090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2465,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5048,"num_write_kb":20176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1015,"num_bytes_recovered":4149248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2465},{"shards":"2","objects":2579}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5052","reported_seq":5128,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.464854+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:46.464854+0000","last_peered":"2024-09-18T11:55:46.464854+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:46.464854+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20661458,"num_objects":5046,"num_object_clones":0,"num_object_copies":10092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5050,"num_write_kb":20180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1514},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5138","reported_seq":5192,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:47.561983+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:47.561983+0000","last_peered":"2024-09-18T11:55:47.561983+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:47.561983+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5138,"log_dups_size":0,"ondisk_log_size":5138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21026002,"num_objects":5135,"num_object_clones":0,"num_object_copies":10270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5136,"num_write_kb":20535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1585}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5124","reported_seq":5331,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:47.542284+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:47.542284+0000","last_peered":"2024-09-18T11:55:47.542284+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:47.542284+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20963328,"num_objects":5118,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1604,"num_write_kb":6416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1525}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5052","reported_seq":5131,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.417144+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:46.417144+0000","last_peered":"2024-09-18T11:55:46.417144+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:46.417144+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20685417,"num_objects":5051,"num_object_clones":0,"num_object_copies":10102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":20201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1516},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5114","reported_seq":5196,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.421013+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:46.421013+0000","last_peered":"2024-09-18T11:55:46.421013+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:46.421013+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5114,"log_dups_size":0,"ondisk_log_size":5114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20916027,"num_objects":5109,"num_object_clones":0,"num_object_copies":10218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5111,"num_write_kb":20428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1587},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5046","reported_seq":5125,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.469592+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:46.469592+0000","last_peered":"2024-09-18T11:55:46.469592+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:46.469592+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20653266,"num_objects":5044,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":20170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1451},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5100","reported_seq":5183,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:46.468241+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:46.468241+0000","last_peered":"2024-09-18T11:55:46.468241+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:46.468241+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20842916,"num_objects":5092,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5097,"num_write_kb":20359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1569},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":333066292,"num_objects":81332,"num_object_clones":0,"num_object_copies":162664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2465,"num_objects_unfound":0,"num_objects_dirty":81332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65387,"num_write_kb":261428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1035,"num_bytes_recovered":4158057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":669888512,"data_stored":669801537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81394,"ondisk_log_size":81394,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051607,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586848,"kb_used_data":276504,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93784992,"statfs":{"total":96636764160,"available":96035831808,"internally_reserved":0,"allocated":283140096,"data_stored":282926940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.56999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]}]},{"osd":1,"up_from":35,"seq":150323855370,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381920,"kb_used_data":75736,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93989920,"statfs":{"total":96636764160,"available":96245678080,"internally_reserved":0,"allocated":77553664,"data_stored":77260348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.214}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601984,"kb_used_data":300024,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93769856,"statfs":{"total":96636764160,"available":96020332544,"internally_reserved":0,"allocated":307224576,"data_stored":307007798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":307130368,"data_stored":307126889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78950400,"data_stored":78870383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283807744,"data_stored":283804265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:51.113 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 85 20 83606 83586 3.8403 2.125 0.0318659 0.0203349 2024-09-18T11:55:52.113 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 86 20 84174 84154 3.82144 2.21875 0.0177281 0.0204352 2024-09-18T11:55:53.115 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 87 20 84740 84720 3.80286 2.21094 0.0104418 0.0205364 2024-09-18T11:55:53.539 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:54.115 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 88 20 85271 85251 3.78322 2.07422 0.0140498 0.0206409 2024-09-18T11:55:54.163 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:54.182 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:54.190 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-18T11:55:53.051885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":344330311,"num_objects":84084,"num_object_clones":0,"num_object_copies":168168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2339,"num_objects_unfound":0,"num_objects_dirty":84084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68139,"num_write_kb":272430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":4674172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84147,"ondisk_log_size":84147,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1578360,"kb_used_data":676512,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281537160,"statfs":{"total":289910292480,"available":288294051840,"internally_reserved":0,"allocated":692748288,"data_stored":692025038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,7,13,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34013184,"num_objects":8304,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-360,"num_objects_unfound":0,"num_objects_dirty":8304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8304,"num_write_kb":33216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":360,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004332"},"pg_stats":[{"pgid":"2.f","version":"37'5319","reported_seq":5388,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.461804+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:51.461804+0000","last_peered":"2024-09-18T11:55:51.461804+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:51.461804+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21786624,"num_objects":5319,"num_object_clones":0,"num_object_copies":10638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":21276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1720},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5277","reported_seq":5433,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:52.575546+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:52.575546+0000","last_peered":"2024-09-18T11:55:52.575546+0000","last_clean":"2024-09-18T11:55:52.575546+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:52.575546+0000","last_undegraded":"2024-09-18T11:55:52.575546+0000","last_fullsized":"2024-09-18T11:55:52.575546+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21590016,"num_objects":5271,"num_object_clones":0,"num_object_copies":10542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2469,"num_write_kb":9876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5297","reported_seq":5480,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:52.585800+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:52.585800+0000","last_peered":"2024-09-18T11:55:52.585800+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:52.585800+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21680128,"num_objects":5293,"num_object_clones":0,"num_object_copies":10586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2155,"num_write_kb":8620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5257","reported_seq":5284,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:52.581871+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:52.581871+0000","last_peered":"2024-09-18T11:55:52.581871+0000","last_clean":"2024-09-18T11:55:52.581871+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:52.581871+0000","last_undegraded":"2024-09-18T11:55:52.581871+0000","last_fullsized":"2024-09-18T11:55:52.581871+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21532672,"num_objects":5257,"num_object_clones":0,"num_object_copies":10514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5257,"num_write_kb":21028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5211","reported_seq":5400,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:51.469336+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:51.469336+0000","last_peered":"2024-09-18T11:55:51.469336+0000","last_clean":"2024-09-18T11:55:51.469336+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:51.469336+0000","last_undegraded":"2024-09-18T11:55:51.469336+0000","last_fullsized":"2024-09-18T11:55:51.469336+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21315584,"num_objects":5204,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2376,"num_write_kb":9504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'5154","reported_seq":5228,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.485646+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:51.485646+0000","last_peered":"2024-09-18T11:55:51.485646+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:51.485646+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21091538,"num_objects":5151,"num_object_clones":0,"num_object_copies":10302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5152,"num_write_kb":20599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1624},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5224","reported_seq":5309,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.456754+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:51.456754+0000","last_peered":"2024-09-18T11:55:51.456754+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:51.456754+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21374162,"num_objects":5220,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":20876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1661},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5340","reported_seq":5563,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:51.447174+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:51.447174+0000","last_peered":"2024-09-18T11:55:51.447174+0000","last_clean":"2024-09-18T11:55:51.447174+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:51.447174+0000","last_undegraded":"2024-09-18T11:55:51.447174+0000","last_fullsized":"2024-09-18T11:55:51.447174+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21864448,"num_objects":5338,"num_object_clones":0,"num_object_copies":10676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1657,"num_write_kb":6628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5218","reported_seq":7648,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:52.572876+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:52.572876+0000","last_peered":"2024-09-18T11:55:52.572876+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:52.572876+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5218,"log_dups_size":0,"ondisk_log_size":5218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21345490,"num_objects":5213,"num_object_clones":0,"num_object_copies":10426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2339,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":20848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":4665344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2339},{"shards":"2","objects":2873}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5221","reported_seq":5296,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.391543+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:51.391543+0000","last_peered":"2024-09-18T11:55:51.391543+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:51.391543+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5221,"log_dups_size":0,"ondisk_log_size":5221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21353682,"num_objects":5215,"num_object_clones":0,"num_object_copies":10430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5219,"num_write_kb":20856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1683},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5321","reported_seq":5375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:52.483768+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:52.483768+0000","last_peered":"2024-09-18T11:55:52.483768+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:52.483768+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5321,"log_dups_size":0,"ondisk_log_size":5321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21775570,"num_objects":5318,"num_object_clones":0,"num_object_copies":10636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":21267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1768}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5295","reported_seq":5502,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:52.543658+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:52.543658+0000","last_peered":"2024-09-18T11:55:52.543658+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:52.543658+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21663744,"num_objects":5289,"num_object_clones":0,"num_object_copies":10578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1775,"num_write_kb":7100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5223","reported_seq":5303,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.435545+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:51.435545+0000","last_peered":"2024-09-18T11:55:51.435545+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:51.435545+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21385833,"num_objects":5222,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":20885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1687},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5294","reported_seq":5376,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.432993+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:51.432993+0000","last_peered":"2024-09-18T11:55:51.432993+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:51.432993+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5294,"log_dups_size":0,"ondisk_log_size":5294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21653307,"num_objects":5289,"num_object_clones":0,"num_object_copies":10578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5291,"num_write_kb":21148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1767},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5214","reported_seq":5292,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.487555+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:51.487555+0000","last_peered":"2024-09-18T11:55:51.487555+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:51.487555+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5214,"log_dups_size":0,"ondisk_log_size":5214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21341394,"num_objects":5212,"num_object_clones":0,"num_object_copies":10424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5212,"num_write_kb":20842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1619},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5279","reported_seq":5362,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.490393+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:51.490393+0000","last_peered":"2024-09-18T11:55:51.490393+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:51.490393+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5279,"log_dups_size":0,"ondisk_log_size":5279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21576100,"num_objects":5271,"num_object_clones":0,"num_object_copies":10542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":21075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1748},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":344330292,"num_objects":84082,"num_object_clones":0,"num_object_copies":168164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2339,"num_objects_unfound":0,"num_objects_dirty":84082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68137,"num_write_kb":272428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":4674153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":692854784,"data_stored":692767809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84144,"ondisk_log_size":84144,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051608,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579908,"kb_used_data":282044,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93791932,"statfs":{"total":96636764160,"available":96042938368,"internally_reserved":0,"allocated":288813056,"data_stored":288599900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.22},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":9.2279999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.109}]}]},{"osd":1,"up_from":35,"seq":150323855371,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384068,"kb_used_data":86204,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93987772,"statfs":{"total":96636764160,"available":96243478528,"internally_reserved":0,"allocated":88272896,"data_stored":87979580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4}]}]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614384,"kb_used_data":308264,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93757456,"statfs":{"total":96636764160,"available":96007634944,"internally_reserved":0,"allocated":315662336,"data_stored":315445558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.373}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315506688,"data_stored":315503209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87965696,"data_stored":87885679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289382400,"data_stored":289378921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:54.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:54.995 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:55.035 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:55.085 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-18T11:55:53.051885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":344330311,"num_objects":84084,"num_object_clones":0,"num_object_copies":168168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2339,"num_objects_unfound":0,"num_objects_dirty":84084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68139,"num_write_kb":272430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":4674172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84147,"ondisk_log_size":84147,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1578360,"kb_used_data":676512,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281537160,"statfs":{"total":289910292480,"available":288294051840,"internally_reserved":0,"allocated":692748288,"data_stored":692025038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,7,13,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34013184,"num_objects":8304,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-360,"num_objects_unfound":0,"num_objects_dirty":8304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8304,"num_write_kb":33216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":360,"num_bytes_recovered":1474560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004332"},"pg_stats":[{"pgid":"2.f","version":"37'5319","reported_seq":5388,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.461804+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:51.461804+0000","last_peered":"2024-09-18T11:55:51.461804+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:51.461804+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21786624,"num_objects":5319,"num_object_clones":0,"num_object_copies":10638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":21276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1720},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5277","reported_seq":5433,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:52.575546+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:52.575546+0000","last_peered":"2024-09-18T11:55:52.575546+0000","last_clean":"2024-09-18T11:55:52.575546+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:52.575546+0000","last_undegraded":"2024-09-18T11:55:52.575546+0000","last_fullsized":"2024-09-18T11:55:52.575546+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21590016,"num_objects":5271,"num_object_clones":0,"num_object_copies":10542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2469,"num_write_kb":9876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5297","reported_seq":5480,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:52.585800+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:52.585800+0000","last_peered":"2024-09-18T11:55:52.585800+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:52.585800+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21680128,"num_objects":5293,"num_object_clones":0,"num_object_copies":10586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2155,"num_write_kb":8620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5257","reported_seq":5284,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:52.581871+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:52.581871+0000","last_peered":"2024-09-18T11:55:52.581871+0000","last_clean":"2024-09-18T11:55:52.581871+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:52.581871+0000","last_undegraded":"2024-09-18T11:55:52.581871+0000","last_fullsized":"2024-09-18T11:55:52.581871+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21532672,"num_objects":5257,"num_object_clones":0,"num_object_copies":10514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5257,"num_write_kb":21028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5211","reported_seq":5400,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:51.469336+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:51.469336+0000","last_peered":"2024-09-18T11:55:51.469336+0000","last_clean":"2024-09-18T11:55:51.469336+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:51.469336+0000","last_undegraded":"2024-09-18T11:55:51.469336+0000","last_fullsized":"2024-09-18T11:55:51.469336+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21315584,"num_objects":5204,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2376,"num_write_kb":9504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'5154","reported_seq":5228,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.485646+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:51.485646+0000","last_peered":"2024-09-18T11:55:51.485646+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:51.485646+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21091538,"num_objects":5151,"num_object_clones":0,"num_object_copies":10302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5152,"num_write_kb":20599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1624},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5224","reported_seq":5309,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.456754+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:51.456754+0000","last_peered":"2024-09-18T11:55:51.456754+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:51.456754+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21374162,"num_objects":5220,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":20876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1661},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5340","reported_seq":5563,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:51.447174+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:51.447174+0000","last_peered":"2024-09-18T11:55:51.447174+0000","last_clean":"2024-09-18T11:55:51.447174+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:51.447174+0000","last_undegraded":"2024-09-18T11:55:51.447174+0000","last_fullsized":"2024-09-18T11:55:51.447174+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21864448,"num_objects":5338,"num_object_clones":0,"num_object_copies":10676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1657,"num_write_kb":6628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5218","reported_seq":7648,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:52.572876+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:52.572876+0000","last_peered":"2024-09-18T11:55:52.572876+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:52.572876+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5218,"log_dups_size":0,"ondisk_log_size":5218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21345490,"num_objects":5213,"num_object_clones":0,"num_object_copies":10426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2339,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":20848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":4665344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2339},{"shards":"2","objects":2873}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5221","reported_seq":5296,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.391543+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:51.391543+0000","last_peered":"2024-09-18T11:55:51.391543+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:51.391543+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5221,"log_dups_size":0,"ondisk_log_size":5221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21353682,"num_objects":5215,"num_object_clones":0,"num_object_copies":10430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5219,"num_write_kb":20856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1683},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5321","reported_seq":5375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:52.483768+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:52.483768+0000","last_peered":"2024-09-18T11:55:52.483768+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:52.483768+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5321,"log_dups_size":0,"ondisk_log_size":5321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21775570,"num_objects":5318,"num_object_clones":0,"num_object_copies":10636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":21267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1768}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5295","reported_seq":5502,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:52.543658+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:52.543658+0000","last_peered":"2024-09-18T11:55:52.543658+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:52.543658+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21663744,"num_objects":5289,"num_object_clones":0,"num_object_copies":10578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1775,"num_write_kb":7100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5223","reported_seq":5303,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.435545+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:51.435545+0000","last_peered":"2024-09-18T11:55:51.435545+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:51.435545+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21385833,"num_objects":5222,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":20885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1687},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5294","reported_seq":5376,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.432993+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:51.432993+0000","last_peered":"2024-09-18T11:55:51.432993+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:51.432993+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5294,"log_dups_size":0,"ondisk_log_size":5294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21653307,"num_objects":5289,"num_object_clones":0,"num_object_copies":10578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5291,"num_write_kb":21148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1767},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5214","reported_seq":5292,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.487555+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:51.487555+0000","last_peered":"2024-09-18T11:55:51.487555+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:51.487555+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5214,"log_dups_size":0,"ondisk_log_size":5214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21341394,"num_objects":5212,"num_object_clones":0,"num_object_copies":10424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5212,"num_write_kb":20842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1619},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5279","reported_seq":5362,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.490393+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:51.490393+0000","last_peered":"2024-09-18T11:55:51.490393+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:51.490393+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5279,"log_dups_size":0,"ondisk_log_size":5279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21576100,"num_objects":5271,"num_object_clones":0,"num_object_copies":10542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":21075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1748},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":344330292,"num_objects":84082,"num_object_clones":0,"num_object_copies":168164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2339,"num_objects_unfound":0,"num_objects_dirty":84082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68137,"num_write_kb":272428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":4674153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":692854784,"data_stored":692767809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84144,"ondisk_log_size":84144,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051608,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579908,"kb_used_data":282044,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93791932,"statfs":{"total":96636764160,"available":96042938368,"internally_reserved":0,"allocated":288813056,"data_stored":288599900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.22},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":9.2279999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.109}]}]},{"osd":1,"up_from":35,"seq":150323855371,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384068,"kb_used_data":86204,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93987772,"statfs":{"total":96636764160,"available":96243478528,"internally_reserved":0,"allocated":88272896,"data_stored":87979580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4}]}]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614384,"kb_used_data":308264,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93757456,"statfs":{"total":96636764160,"available":96007634944,"internally_reserved":0,"allocated":315662336,"data_stored":315445558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.373}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315506688,"data_stored":315503209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87965696,"data_stored":87885679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289382400,"data_stored":289378921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:55:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'5319", 'reported_seq': 5388, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:51.461804+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:55:51.461804+0000', 'last_peered': '2024-09-18T11:55:51.461804+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:55:51.461804+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5319, 'log_dups_size': 0, 'ondisk_log_size': 5319, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21786624, 'num_objects': 5319, 'num_object_clones': 0, 'num_object_copies': 10638, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5319, 'num_write_kb': 21276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1720}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:55:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'5297", 'reported_seq': 5480, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:52.585800+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:55:52.585800+0000', 'last_peered': '2024-09-18T11:55:52.585800+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:55:52.585800+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 5297, 'log_dups_size': 0, 'ondisk_log_size': 5297, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21680128, 'num_objects': 5293, 'num_object_clones': 0, 'num_object_copies': 10586, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2155, 'num_write_kb': 8620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 1757}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:55:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'5154", 'reported_seq': 5228, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:51.485646+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:55:51.485646+0000', 'last_peered': '2024-09-18T11:55:51.485646+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:55:51.485646+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5154, 'log_dups_size': 0, 'ondisk_log_size': 5154, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21091538, 'num_objects': 5151, 'num_object_clones': 0, 'num_object_copies': 10302, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5152, 'num_write_kb': 20599, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1624}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:55:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'5224", 'reported_seq': 5309, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:51.456754+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:55:51.456754+0000', 'last_peered': '2024-09-18T11:55:51.456754+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:55:51.456754+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5224, 'log_dups_size': 0, 'ondisk_log_size': 5224, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21374162, 'num_objects': 5220, 'num_object_clones': 0, 'num_object_copies': 10440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5222, 'num_write_kb': 20876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1661}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:55:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'5218", 'reported_seq': 7648, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:52.572876+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:55:52.572876+0000', 'last_peered': '2024-09-18T11:55:52.572876+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:55:52.572876+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 5218, 'log_dups_size': 0, 'ondisk_log_size': 5218, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21345490, 'num_objects': 5213, 'num_object_clones': 0, 'num_object_copies': 10426, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 2339, 'num_objects_unfound': 0, 'num_objects_dirty': 5213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5216, 'num_write_kb': 20848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1141, 'num_bytes_recovered': 4665344, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2339}, {'shards': '2', 'objects': 2873}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:55:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'5221", 'reported_seq': 5296, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:51.391543+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:55:51.391543+0000', 'last_peered': '2024-09-18T11:55:51.391543+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:55:51.391543+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5221, 'log_dups_size': 0, 'ondisk_log_size': 5221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21353682, 'num_objects': 5215, 'num_object_clones': 0, 'num_object_copies': 10430, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5215, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5219, 'num_write_kb': 20856, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1683}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:55:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'5321", 'reported_seq': 5375, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:52.483768+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:55:52.483768+0000', 'last_peered': '2024-09-18T11:55:52.483768+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:55:52.483768+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5321, 'log_dups_size': 0, 'ondisk_log_size': 5321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21775570, 'num_objects': 5318, 'num_object_clones': 0, 'num_object_copies': 10636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5319, 'num_write_kb': 21267, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 1768}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:55.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:55:55.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'5295", 'reported_seq': 5502, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:52.543658+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:55:52.543658+0000', 'last_peered': '2024-09-18T11:55:52.543658+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:55:52.543658+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 5295, 'log_dups_size': 0, 'ondisk_log_size': 5295, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21663744, 'num_objects': 5289, 'num_object_clones': 0, 'num_object_copies': 10578, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1775, 'num_write_kb': 7100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 1696}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:55:55.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:55:55.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'5223", 'reported_seq': 5303, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:51.435545+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:55:51.435545+0000', 'last_peered': '2024-09-18T11:55:51.435545+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:55:51.435545+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5223, 'log_dups_size': 0, 'ondisk_log_size': 5223, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21385833, 'num_objects': 5222, 'num_object_clones': 0, 'num_object_copies': 10444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5222, 'num_write_kb': 20885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1687}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:55.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:55:55.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'5294", 'reported_seq': 5376, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:51.432993+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:55:51.432993+0000', 'last_peered': '2024-09-18T11:55:51.432993+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:55:51.432993+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5294, 'log_dups_size': 0, 'ondisk_log_size': 5294, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21653307, 'num_objects': 5289, 'num_object_clones': 0, 'num_object_copies': 10578, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5291, 'num_write_kb': 21148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1767}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:55.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:55:55.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'5214", 'reported_seq': 5292, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:51.487555+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:55:51.487555+0000', 'last_peered': '2024-09-18T11:55:51.487555+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:55:51.487555+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5214, 'log_dups_size': 0, 'ondisk_log_size': 5214, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21341394, 'num_objects': 5212, 'num_object_clones': 0, 'num_object_copies': 10424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5212, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5212, 'num_write_kb': 20842, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1619}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:55.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:55:55.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'5279", 'reported_seq': 5362, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:51.490393+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:55:51.490393+0000', 'last_peered': '2024-09-18T11:55:51.490393+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:55:51.490393+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5279, 'log_dups_size': 0, 'ondisk_log_size': 5279, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21576100, 'num_objects': 5271, 'num_object_clones': 0, 'num_object_copies': 10542, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5271, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5276, 'num_write_kb': 21075, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1748}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:55:55.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:55:55.118 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 89 20 85812 85792 3.76429 2.11328 0.0229657 0.0207472 2024-09-18T11:55:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:55.928 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/168168 objects degraded (23.245%), 12 pgs degraded","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":155,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":84084,"data_bytes":344330311,"bytes_used":1616240640,"bytes_avail":288294051840,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":168168,"degraded_ratio":0.23244612530326816,"misplaced_objects":2339,"misplaced_total":168168,"misplaced_ratio":0.013908710337281766,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":82572,"recovering_keys_per_sec":0,"num_objects_recovered":242,"num_bytes_recovered":991232,"num_keys_recovered":0,"write_bytes_sec":1869140,"read_op_per_sec":0,"write_op_per_sec":456},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:55:19.042648+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:55:55.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:55:55.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:55:56.119 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 90 20 86330 86310 3.74495 2.02344 0.0691005 0.0208516 2024-09-18T11:55:56.725 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:55:56.734 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:55:56.759 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-09-18T11:55:55.052356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":344330311,"num_objects":84084,"num_object_clones":0,"num_object_copies":168168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2339,"num_objects_unfound":0,"num_objects_dirty":84084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68139,"num_write_kb":272430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":4674172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84147,"ondisk_log_size":84147,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1578360,"kb_used_data":676512,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281537160,"statfs":{"total":289910292480,"available":288294051840,"internally_reserved":0,"allocated":692748288,"data_stored":692025038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,7,13,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22437888,"num_objects":5478,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-242,"num_objects_unfound":0,"num_objects_dirty":5478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5478,"num_write_kb":21912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004387"},"pg_stats":[{"pgid":"2.f","version":"37'5319","reported_seq":5388,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.461804+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:51.461804+0000","last_peered":"2024-09-18T11:55:51.461804+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:51.461804+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21786624,"num_objects":5319,"num_object_clones":0,"num_object_copies":10638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":21276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1720},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5277","reported_seq":5433,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:52.575546+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:52.575546+0000","last_peered":"2024-09-18T11:55:52.575546+0000","last_clean":"2024-09-18T11:55:52.575546+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:52.575546+0000","last_undegraded":"2024-09-18T11:55:52.575546+0000","last_fullsized":"2024-09-18T11:55:52.575546+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21590016,"num_objects":5271,"num_object_clones":0,"num_object_copies":10542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2469,"num_write_kb":9876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5297","reported_seq":5480,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:52.585800+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:52.585800+0000","last_peered":"2024-09-18T11:55:52.585800+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:52.585800+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21680128,"num_objects":5293,"num_object_clones":0,"num_object_copies":10586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2155,"num_write_kb":8620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5257","reported_seq":5284,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:52.581871+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:52.581871+0000","last_peered":"2024-09-18T11:55:52.581871+0000","last_clean":"2024-09-18T11:55:52.581871+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:52.581871+0000","last_undegraded":"2024-09-18T11:55:52.581871+0000","last_fullsized":"2024-09-18T11:55:52.581871+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21532672,"num_objects":5257,"num_object_clones":0,"num_object_copies":10514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5257,"num_write_kb":21028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5211","reported_seq":5400,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:51.469336+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:51.469336+0000","last_peered":"2024-09-18T11:55:51.469336+0000","last_clean":"2024-09-18T11:55:51.469336+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:51.469336+0000","last_undegraded":"2024-09-18T11:55:51.469336+0000","last_fullsized":"2024-09-18T11:55:51.469336+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21315584,"num_objects":5204,"num_object_clones":0,"num_object_copies":10408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2376,"num_write_kb":9504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'5154","reported_seq":5228,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.485646+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:51.485646+0000","last_peered":"2024-09-18T11:55:51.485646+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:51.485646+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21091538,"num_objects":5151,"num_object_clones":0,"num_object_copies":10302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5152,"num_write_kb":20599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1624},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5224","reported_seq":5309,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.456754+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:51.456754+0000","last_peered":"2024-09-18T11:55:51.456754+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:51.456754+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21374162,"num_objects":5220,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":20876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1661},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5340","reported_seq":5563,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:51.447174+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:51.447174+0000","last_peered":"2024-09-18T11:55:51.447174+0000","last_clean":"2024-09-18T11:55:51.447174+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:51.447174+0000","last_undegraded":"2024-09-18T11:55:51.447174+0000","last_fullsized":"2024-09-18T11:55:51.447174+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21864448,"num_objects":5338,"num_object_clones":0,"num_object_copies":10676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1657,"num_write_kb":6628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5218","reported_seq":7648,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:52.572876+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:52.572876+0000","last_peered":"2024-09-18T11:55:52.572876+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:52.572876+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5218,"log_dups_size":0,"ondisk_log_size":5218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21345490,"num_objects":5213,"num_object_clones":0,"num_object_copies":10426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2339,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":20848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":4665344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2339},{"shards":"2","objects":2873}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5221","reported_seq":5296,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.391543+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:51.391543+0000","last_peered":"2024-09-18T11:55:51.391543+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:51.391543+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5221,"log_dups_size":0,"ondisk_log_size":5221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21353682,"num_objects":5215,"num_object_clones":0,"num_object_copies":10430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5219,"num_write_kb":20856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1683},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5321","reported_seq":5375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:52.483768+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:52.483768+0000","last_peered":"2024-09-18T11:55:52.483768+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:52.483768+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5321,"log_dups_size":0,"ondisk_log_size":5321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21775570,"num_objects":5318,"num_object_clones":0,"num_object_copies":10636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":21267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1768}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5295","reported_seq":5502,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:52.543658+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:52.543658+0000","last_peered":"2024-09-18T11:55:52.543658+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:52.543658+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21663744,"num_objects":5289,"num_object_clones":0,"num_object_copies":10578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1775,"num_write_kb":7100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5223","reported_seq":5303,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.435545+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:51.435545+0000","last_peered":"2024-09-18T11:55:51.435545+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:51.435545+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21385833,"num_objects":5222,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":20885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1687},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5294","reported_seq":5376,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.432993+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:51.432993+0000","last_peered":"2024-09-18T11:55:51.432993+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:51.432993+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5294,"log_dups_size":0,"ondisk_log_size":5294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21653307,"num_objects":5289,"num_object_clones":0,"num_object_copies":10578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5291,"num_write_kb":21148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1767},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5214","reported_seq":5292,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.487555+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:51.487555+0000","last_peered":"2024-09-18T11:55:51.487555+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:51.487555+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5214,"log_dups_size":0,"ondisk_log_size":5214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21341394,"num_objects":5212,"num_object_clones":0,"num_object_copies":10424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5212,"num_write_kb":20842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1619},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5279","reported_seq":5362,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:51.490393+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:51.490393+0000","last_peered":"2024-09-18T11:55:51.490393+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:51.490393+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5279,"log_dups_size":0,"ondisk_log_size":5279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21576100,"num_objects":5271,"num_object_clones":0,"num_object_copies":10542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":21075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1748},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":344330292,"num_objects":84082,"num_object_clones":0,"num_object_copies":168164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2339,"num_objects_unfound":0,"num_objects_dirty":84082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68137,"num_write_kb":272428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":4674153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":692854784,"data_stored":692767809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84144,"ondisk_log_size":84144,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051608,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579908,"kb_used_data":282044,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93791932,"statfs":{"total":96636764160,"available":96042938368,"internally_reserved":0,"allocated":288813056,"data_stored":288599900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.22},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":9.2279999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.109}]}]},{"osd":1,"up_from":35,"seq":150323855371,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384068,"kb_used_data":86204,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93987772,"statfs":{"total":96636764160,"available":96243478528,"internally_reserved":0,"allocated":88272896,"data_stored":87979580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4}]}]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614384,"kb_used_data":308264,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93757456,"statfs":{"total":96636764160,"available":96007634944,"internally_reserved":0,"allocated":315662336,"data_stored":315445558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.373}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315506688,"data_stored":315503209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87965696,"data_stored":87885679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289382400,"data_stored":289378921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:55:57.120 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 91 20 86863 86843 3.72668 2.08203 0.0338677 0.0209546 2024-09-18T11:55:58.119 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 92 20 87436 87416 3.71051 2.23828 0.0140206 0.0210462 2024-09-18T11:55:59.119 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 93 20 87979 87959 3.69342 2.12109 0.0408394 0.0211436 2024-09-18T11:55:59.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:00.119 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 94 20 88518 88498 3.67653 2.10547 0.0379844 0.0212387 2024-09-18T11:56:00.452 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:00.462 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:00.471 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-18T11:55:59.053196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":355364935,"num_objects":86778,"num_object_clones":0,"num_object_copies":173556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2206,"num_objects_unfound":0,"num_objects_dirty":86778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70833,"num_write_kb":283206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1295,"num_bytes_recovered":5218940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86841,"ondisk_log_size":86841,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1598836,"kb_used_data":696988,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281516684,"statfs":{"total":289910292480,"available":288273084416,"internally_reserved":0,"allocated":713715712,"data_stored":712992462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,8,3,4,16,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26558464,"num_objects":6484,"num_object_clones":0,"num_object_copies":12968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-375,"num_objects_unfound":0,"num_objects_dirty":6484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6484,"num_write_kb":25936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":375,"num_bytes_recovered":1536000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002583"},"pg_stats":[{"pgid":"2.f","version":"37'5490","reported_seq":5559,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.429642+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:56.429642+0000","last_peered":"2024-09-18T11:55:56.429642+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:56.429642+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5490,"log_dups_size":0,"ondisk_log_size":5490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22487040,"num_objects":5490,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5490,"num_write_kb":21960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1891},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5439","reported_seq":5595,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.501874+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:57.501874+0000","last_peered":"2024-09-18T11:55:57.501874+0000","last_clean":"2024-09-18T11:55:57.501874+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:57.501874+0000","last_undegraded":"2024-09-18T11:55:57.501874+0000","last_fullsized":"2024-09-18T11:55:57.501874+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5439,"log_dups_size":0,"ondisk_log_size":5439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22253568,"num_objects":5433,"num_object_clones":0,"num_object_copies":10866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2631,"num_write_kb":10524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5449","reported_seq":5632,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:57.471517+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:57.471517+0000","last_peered":"2024-09-18T11:55:57.471517+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:57.471517+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5449,"log_dups_size":0,"ondisk_log_size":5449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22302720,"num_objects":5445,"num_object_clones":0,"num_object_copies":10890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2307,"num_write_kb":9228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5431","reported_seq":5458,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.579872+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:57.579872+0000","last_peered":"2024-09-18T11:55:57.579872+0000","last_clean":"2024-09-18T11:55:57.579872+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:57.579872+0000","last_undegraded":"2024-09-18T11:55:57.579872+0000","last_fullsized":"2024-09-18T11:55:57.579872+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5431,"log_dups_size":0,"ondisk_log_size":5431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22245376,"num_objects":5431,"num_object_clones":0,"num_object_copies":10862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5431,"num_write_kb":21724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5382","reported_seq":5569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:56.470138+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:56.470138+0000","last_peered":"2024-09-18T11:55:56.470138+0000","last_clean":"2024-09-18T11:55:56.470138+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:56.470138+0000","last_undegraded":"2024-09-18T11:55:56.470138+0000","last_fullsized":"2024-09-18T11:55:56.470138+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22016000,"num_objects":5375,"num_object_clones":0,"num_object_copies":10750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2547,"num_write_kb":10188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'5331","reported_seq":5410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.519742+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:56.519742+0000","last_peered":"2024-09-18T11:55:56.519742+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:56.519742+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21816530,"num_objects":5328,"num_object_clones":0,"num_object_copies":10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5329,"num_write_kb":21307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1801},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5397","reported_seq":5482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.507108+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:56.507108+0000","last_peered":"2024-09-18T11:55:56.507108+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:56.507108+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22082770,"num_objects":5393,"num_object_clones":0,"num_object_copies":10786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5395,"num_write_kb":21568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1834},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5512","reported_seq":5735,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:56.459726+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:56.459726+0000","last_peered":"2024-09-18T11:55:56.459726+0000","last_clean":"2024-09-18T11:55:56.459726+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:56.459726+0000","last_undegraded":"2024-09-18T11:55:56.459726+0000","last_fullsized":"2024-09-18T11:55:56.459726+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22568960,"num_objects":5510,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1829,"num_write_kb":7316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5379","reported_seq":8073,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:57.590644+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:57.590644+0000","last_peered":"2024-09-18T11:55:57.590644+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:57.590644+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5379,"log_dups_size":0,"ondisk_log_size":5379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22004946,"num_objects":5374,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2206,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5377,"num_write_kb":21492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":5210112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2206},{"shards":"2","objects":3167}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5407","reported_seq":5481,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.536291+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:56.536291+0000","last_peered":"2024-09-18T11:55:56.536291+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:56.536291+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22115538,"num_objects":5401,"num_object_clones":0,"num_object_copies":10802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5405,"num_write_kb":21600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1869},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5487","reported_seq":5539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:57.590600+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:57.590600+0000","last_peered":"2024-09-18T11:55:57.590600+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:57.590600+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5487,"log_dups_size":0,"ondisk_log_size":5487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22455506,"num_objects":5484,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":21931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5473","reported_seq":5680,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:57.577593+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:57.577593+0000","last_peered":"2024-09-18T11:55:57.577593+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:57.577593+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5473,"log_dups_size":0,"ondisk_log_size":5473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22392832,"num_objects":5467,"num_object_clones":0,"num_object_copies":10934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1953,"num_write_kb":7812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1874}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5382","reported_seq":5462,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.463371+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:56.463371+0000","last_peered":"2024-09-18T11:55:56.463371+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:56.463371+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22037097,"num_objects":5381,"num_object_clones":0,"num_object_copies":10762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5381,"num_write_kb":21521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1846},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5462","reported_seq":5544,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.485776+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:56.485776+0000","last_peered":"2024-09-18T11:55:56.485776+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:56.485776+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5462,"log_dups_size":0,"ondisk_log_size":5462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22341435,"num_objects":5457,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5459,"num_write_kb":21820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1935},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5382","reported_seq":5461,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.489426+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:56.489426+0000","last_peered":"2024-09-18T11:55:56.489426+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:56.489426+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22029522,"num_objects":5380,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5380,"num_write_kb":21514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1787},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5435","reported_seq":5516,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.528481+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:56.528481+0000","last_peered":"2024-09-18T11:55:56.528481+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:56.528481+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5435,"log_dups_size":0,"ondisk_log_size":5435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22215076,"num_objects":5427,"num_object_clones":0,"num_object_copies":10854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5432,"num_write_kb":21699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1904},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":355364916,"num_objects":86776,"num_object_clones":0,"num_object_copies":173552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2206,"num_objects_unfound":0,"num_objects_dirty":86776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70831,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":5218921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715501568,"data_stored":715414593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86838,"ondisk_log_size":86838,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051609,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589372,"kb_used_data":287348,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93782468,"statfs":{"total":96636764160,"available":96033247232,"internally_reserved":0,"allocated":294244352,"data_stored":294031196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.317},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":7.2619999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3239999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855372,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395304,"kb_used_data":93280,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93976536,"statfs":{"total":96636764160,"available":96231972864,"internally_reserved":0,"allocated":95518720,"data_stored":95225404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,3,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614160,"kb_used_data":316360,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93757680,"statfs":{"total":96636764160,"available":96007864320,"internally_reserved":0,"allocated":323952640,"data_stored":323735862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,2,7,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6840000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":2.7639999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":4.7140000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323887104,"data_stored":323883625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96833536,"data_stored":96753519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294780928,"data_stored":294777449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:00.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:01.119 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 95 20 89021 89001 3.65851 1.96484 0.0215549 0.021345 2024-09-18T11:56:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:01.170 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:01.181 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-09-18T11:56:01.053621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":355364935,"num_objects":86778,"num_object_clones":0,"num_object_copies":173556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2206,"num_objects_unfound":0,"num_objects_dirty":86778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70833,"num_write_kb":283206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1295,"num_bytes_recovered":5218940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86841,"ondisk_log_size":86841,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1598836,"kb_used_data":696988,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281516684,"statfs":{"total":289910292480,"available":288273084416,"internally_reserved":0,"allocated":713715712,"data_stored":712992462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,8,3,4,16,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22298624,"num_objects":5444,"num_object_clones":0,"num_object_copies":10888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-259,"num_objects_unfound":0,"num_objects_dirty":5444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5444,"num_write_kb":21776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":1060864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002539"},"pg_stats":[{"pgid":"2.f","version":"37'5490","reported_seq":5559,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.429642+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:56.429642+0000","last_peered":"2024-09-18T11:55:56.429642+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:56.429642+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5490,"log_dups_size":0,"ondisk_log_size":5490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22487040,"num_objects":5490,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5490,"num_write_kb":21960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1891},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5439","reported_seq":5595,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.501874+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:57.501874+0000","last_peered":"2024-09-18T11:55:57.501874+0000","last_clean":"2024-09-18T11:55:57.501874+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:57.501874+0000","last_undegraded":"2024-09-18T11:55:57.501874+0000","last_fullsized":"2024-09-18T11:55:57.501874+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5439,"log_dups_size":0,"ondisk_log_size":5439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22253568,"num_objects":5433,"num_object_clones":0,"num_object_copies":10866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2631,"num_write_kb":10524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5449","reported_seq":5632,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:57.471517+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:57.471517+0000","last_peered":"2024-09-18T11:55:57.471517+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:57.471517+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5449,"log_dups_size":0,"ondisk_log_size":5449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22302720,"num_objects":5445,"num_object_clones":0,"num_object_copies":10890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2307,"num_write_kb":9228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5431","reported_seq":5458,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.579872+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:57.579872+0000","last_peered":"2024-09-18T11:55:57.579872+0000","last_clean":"2024-09-18T11:55:57.579872+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:57.579872+0000","last_undegraded":"2024-09-18T11:55:57.579872+0000","last_fullsized":"2024-09-18T11:55:57.579872+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5431,"log_dups_size":0,"ondisk_log_size":5431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22245376,"num_objects":5431,"num_object_clones":0,"num_object_copies":10862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5431,"num_write_kb":21724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5382","reported_seq":5569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:56.470138+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:56.470138+0000","last_peered":"2024-09-18T11:55:56.470138+0000","last_clean":"2024-09-18T11:55:56.470138+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:56.470138+0000","last_undegraded":"2024-09-18T11:55:56.470138+0000","last_fullsized":"2024-09-18T11:55:56.470138+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22016000,"num_objects":5375,"num_object_clones":0,"num_object_copies":10750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2547,"num_write_kb":10188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'5331","reported_seq":5410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.519742+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:56.519742+0000","last_peered":"2024-09-18T11:55:56.519742+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:56.519742+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21816530,"num_objects":5328,"num_object_clones":0,"num_object_copies":10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5329,"num_write_kb":21307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1801},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5397","reported_seq":5482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.507108+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:56.507108+0000","last_peered":"2024-09-18T11:55:56.507108+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:56.507108+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22082770,"num_objects":5393,"num_object_clones":0,"num_object_copies":10786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5395,"num_write_kb":21568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1834},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5512","reported_seq":5735,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:56.459726+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:56.459726+0000","last_peered":"2024-09-18T11:55:56.459726+0000","last_clean":"2024-09-18T11:55:56.459726+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:56.459726+0000","last_undegraded":"2024-09-18T11:55:56.459726+0000","last_fullsized":"2024-09-18T11:55:56.459726+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22568960,"num_objects":5510,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1829,"num_write_kb":7316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5379","reported_seq":8073,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:57.590644+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:57.590644+0000","last_peered":"2024-09-18T11:55:57.590644+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:57.590644+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5379,"log_dups_size":0,"ondisk_log_size":5379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22004946,"num_objects":5374,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2206,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5377,"num_write_kb":21492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":5210112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2206},{"shards":"2","objects":3167}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5407","reported_seq":5481,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.536291+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:56.536291+0000","last_peered":"2024-09-18T11:55:56.536291+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:56.536291+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22115538,"num_objects":5401,"num_object_clones":0,"num_object_copies":10802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5405,"num_write_kb":21600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1869},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5487","reported_seq":5539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:57.590600+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:57.590600+0000","last_peered":"2024-09-18T11:55:57.590600+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:57.590600+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5487,"log_dups_size":0,"ondisk_log_size":5487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22455506,"num_objects":5484,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":21931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5473","reported_seq":5680,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:57.577593+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:57.577593+0000","last_peered":"2024-09-18T11:55:57.577593+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:57.577593+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5473,"log_dups_size":0,"ondisk_log_size":5473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22392832,"num_objects":5467,"num_object_clones":0,"num_object_copies":10934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1953,"num_write_kb":7812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1874}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5382","reported_seq":5462,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.463371+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:56.463371+0000","last_peered":"2024-09-18T11:55:56.463371+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:56.463371+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22037097,"num_objects":5381,"num_object_clones":0,"num_object_copies":10762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5381,"num_write_kb":21521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1846},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5462","reported_seq":5544,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.485776+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:56.485776+0000","last_peered":"2024-09-18T11:55:56.485776+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:56.485776+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5462,"log_dups_size":0,"ondisk_log_size":5462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22341435,"num_objects":5457,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5459,"num_write_kb":21820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1935},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5382","reported_seq":5461,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.489426+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:56.489426+0000","last_peered":"2024-09-18T11:55:56.489426+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:56.489426+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22029522,"num_objects":5380,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5380,"num_write_kb":21514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1787},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5435","reported_seq":5516,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.528481+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:56.528481+0000","last_peered":"2024-09-18T11:55:56.528481+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:56.528481+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5435,"log_dups_size":0,"ondisk_log_size":5435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22215076,"num_objects":5427,"num_object_clones":0,"num_object_copies":10854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5432,"num_write_kb":21699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1904},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":355364916,"num_objects":86776,"num_object_clones":0,"num_object_copies":173552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2206,"num_objects_unfound":0,"num_objects_dirty":86776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70831,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":5218921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715501568,"data_stored":715414593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86838,"ondisk_log_size":86838,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051609,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589372,"kb_used_data":287348,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93782468,"statfs":{"total":96636764160,"available":96033247232,"internally_reserved":0,"allocated":294244352,"data_stored":294031196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.317},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":7.2619999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3239999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855372,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395304,"kb_used_data":93280,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93976536,"statfs":{"total":96636764160,"available":96231972864,"internally_reserved":0,"allocated":95518720,"data_stored":95225404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,3,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614160,"kb_used_data":316360,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93757680,"statfs":{"total":96636764160,"available":96007864320,"internally_reserved":0,"allocated":323952640,"data_stored":323735862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,2,7,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6840000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":2.7639999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":4.7140000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323887104,"data_stored":323883625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96833536,"data_stored":96753519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294780928,"data_stored":294777449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:01.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:56:01.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'5490", 'reported_seq': 5559, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:56.429642+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:55:56.429642+0000', 'last_peered': '2024-09-18T11:55:56.429642+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:55:56.429642+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5490, 'log_dups_size': 0, 'ondisk_log_size': 5490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22487040, 'num_objects': 5490, 'num_object_clones': 0, 'num_object_copies': 10980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5490, 'num_write_kb': 21960, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1891}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:01.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:56:01.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'5449", 'reported_seq': 5632, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:57.471517+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:55:57.471517+0000', 'last_peered': '2024-09-18T11:55:57.471517+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:55:57.471517+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 5449, 'log_dups_size': 0, 'ondisk_log_size': 5449, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22302720, 'num_objects': 5445, 'num_object_clones': 0, 'num_object_copies': 10890, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2307, 'num_write_kb': 9228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 1909}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:01.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:56:01.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'5331", 'reported_seq': 5410, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:56.519742+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:55:56.519742+0000', 'last_peered': '2024-09-18T11:55:56.519742+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:55:56.519742+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5331, 'log_dups_size': 0, 'ondisk_log_size': 5331, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21816530, 'num_objects': 5328, 'num_object_clones': 0, 'num_object_copies': 10656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5329, 'num_write_kb': 21307, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1801}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:01.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:56:01.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'5397", 'reported_seq': 5482, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:56.507108+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:55:56.507108+0000', 'last_peered': '2024-09-18T11:55:56.507108+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:55:56.507108+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5397, 'log_dups_size': 0, 'ondisk_log_size': 5397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22082770, 'num_objects': 5393, 'num_object_clones': 0, 'num_object_copies': 10786, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5395, 'num_write_kb': 21568, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1834}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:01.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:56:01.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'5379", 'reported_seq': 8073, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:57.590644+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:55:57.590644+0000', 'last_peered': '2024-09-18T11:55:57.590644+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:55:57.590644+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 5379, 'log_dups_size': 0, 'ondisk_log_size': 5379, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22004946, 'num_objects': 5374, 'num_object_clones': 0, 'num_object_copies': 10748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 2206, 'num_objects_unfound': 0, 'num_objects_dirty': 5374, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5377, 'num_write_kb': 21492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1274, 'num_bytes_recovered': 5210112, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2206}, {'shards': '2', 'objects': 3167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:01.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:56:01.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'5407", 'reported_seq': 5481, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:56.536291+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:55:56.536291+0000', 'last_peered': '2024-09-18T11:55:56.536291+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:55:56.536291+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5407, 'log_dups_size': 0, 'ondisk_log_size': 5407, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22115538, 'num_objects': 5401, 'num_object_clones': 0, 'num_object_copies': 10802, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5401, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5405, 'num_write_kb': 21600, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1869}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:01.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:56:01.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'5487", 'reported_seq': 5539, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:57.590600+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:55:57.590600+0000', 'last_peered': '2024-09-18T11:55:57.590600+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:55:57.590600+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5487, 'log_dups_size': 0, 'ondisk_log_size': 5487, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22455506, 'num_objects': 5484, 'num_object_clones': 0, 'num_object_copies': 10968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5485, 'num_write_kb': 21931, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 1934}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:01.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:56:01.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'5473", 'reported_seq': 5680, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:57.577593+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:55:57.577593+0000', 'last_peered': '2024-09-18T11:55:57.577593+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:55:57.577593+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 5473, 'log_dups_size': 0, 'ondisk_log_size': 5473, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22392832, 'num_objects': 5467, 'num_object_clones': 0, 'num_object_copies': 10934, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1953, 'num_write_kb': 7812, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 1874}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:01.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:56:01.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'5382", 'reported_seq': 5462, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:56.463371+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:55:56.463371+0000', 'last_peered': '2024-09-18T11:55:56.463371+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:55:56.463371+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5382, 'log_dups_size': 0, 'ondisk_log_size': 5382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22037097, 'num_objects': 5381, 'num_object_clones': 0, 'num_object_copies': 10762, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5381, 'num_write_kb': 21521, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1846}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:01.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:56:01.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'5462", 'reported_seq': 5544, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:56.485776+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:55:56.485776+0000', 'last_peered': '2024-09-18T11:55:56.485776+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:55:56.485776+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5462, 'log_dups_size': 0, 'ondisk_log_size': 5462, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22341435, 'num_objects': 5457, 'num_object_clones': 0, 'num_object_copies': 10914, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5459, 'num_write_kb': 21820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1935}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:01.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:56:01.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'5382", 'reported_seq': 5461, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:56.489426+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:55:56.489426+0000', 'last_peered': '2024-09-18T11:55:56.489426+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:55:56.489426+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5382, 'log_dups_size': 0, 'ondisk_log_size': 5382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22029522, 'num_objects': 5380, 'num_object_clones': 0, 'num_object_copies': 10760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5380, 'num_write_kb': 21514, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1787}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:01.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:56:01.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'5435", 'reported_seq': 5516, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:55:56.528481+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:55:56.528481+0000', 'last_peered': '2024-09-18T11:55:56.528481+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:55:56.528481+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5435, 'log_dups_size': 0, 'ondisk_log_size': 5435, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22215076, 'num_objects': 5427, 'num_object_clones': 0, 'num_object_copies': 10854, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5432, 'num_write_kb': 21699, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1904}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:01.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:02.045 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:02.045 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/173556 objects degraded (22.523%), 12 pgs degraded","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":162,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":86778,"data_bytes":355364935,"bytes_used":1637208064,"bytes_avail":288273084416,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":173556,"degraded_ratio":0.22522989697849685,"misplaced_objects":2206,"misplaced_total":173556,"misplaced_ratio":0.012710594851229575,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":88386,"recovering_keys_per_sec":0,"num_objects_recovered":259,"num_bytes_recovered":1060864,"num_keys_recovered":0,"write_bytes_sec":1857825,"read_op_per_sec":0,"write_op_per_sec":453},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-18T11:55:19.042648+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T11:56:02.094 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:02.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:02.119 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 96 20 89528 89508 3.64103 1.98047 0.0501377 0.0214494 2024-09-18T11:56:02.636 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:02.644 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:02.664 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-09-18T11:56:01.053621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":355364935,"num_objects":86778,"num_object_clones":0,"num_object_copies":173556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2206,"num_objects_unfound":0,"num_objects_dirty":86778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70833,"num_write_kb":283206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1295,"num_bytes_recovered":5218940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86841,"ondisk_log_size":86841,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1598836,"kb_used_data":696988,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281516684,"statfs":{"total":289910292480,"available":288273084416,"internally_reserved":0,"allocated":713715712,"data_stored":712992462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,8,3,4,16,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22298624,"num_objects":5444,"num_object_clones":0,"num_object_copies":10888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-259,"num_objects_unfound":0,"num_objects_dirty":5444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5444,"num_write_kb":21776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":1060864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002539"},"pg_stats":[{"pgid":"2.f","version":"37'5490","reported_seq":5559,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.429642+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:55:56.429642+0000","last_peered":"2024-09-18T11:55:56.429642+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:55:56.429642+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5490,"log_dups_size":0,"ondisk_log_size":5490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22487040,"num_objects":5490,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5490,"num_write_kb":21960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1891},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5439","reported_seq":5595,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.501874+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:55:57.501874+0000","last_peered":"2024-09-18T11:55:57.501874+0000","last_clean":"2024-09-18T11:55:57.501874+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:55:57.501874+0000","last_undegraded":"2024-09-18T11:55:57.501874+0000","last_fullsized":"2024-09-18T11:55:57.501874+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5439,"log_dups_size":0,"ondisk_log_size":5439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22253568,"num_objects":5433,"num_object_clones":0,"num_object_copies":10866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2631,"num_write_kb":10524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5449","reported_seq":5632,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:57.471517+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:55:57.471517+0000","last_peered":"2024-09-18T11:55:57.471517+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:55:57.471517+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5449,"log_dups_size":0,"ondisk_log_size":5449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22302720,"num_objects":5445,"num_object_clones":0,"num_object_copies":10890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2307,"num_write_kb":9228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":1909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5431","reported_seq":5458,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:57.579872+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:55:57.579872+0000","last_peered":"2024-09-18T11:55:57.579872+0000","last_clean":"2024-09-18T11:55:57.579872+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:55:57.579872+0000","last_undegraded":"2024-09-18T11:55:57.579872+0000","last_fullsized":"2024-09-18T11:55:57.579872+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5431,"log_dups_size":0,"ondisk_log_size":5431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22245376,"num_objects":5431,"num_object_clones":0,"num_object_copies":10862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5431,"num_write_kb":21724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5382","reported_seq":5569,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:56.470138+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:55:56.470138+0000","last_peered":"2024-09-18T11:55:56.470138+0000","last_clean":"2024-09-18T11:55:56.470138+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:55:56.470138+0000","last_undegraded":"2024-09-18T11:55:56.470138+0000","last_fullsized":"2024-09-18T11:55:56.470138+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22016000,"num_objects":5375,"num_object_clones":0,"num_object_copies":10750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2547,"num_write_kb":10188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'5331","reported_seq":5410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.519742+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:55:56.519742+0000","last_peered":"2024-09-18T11:55:56.519742+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:55:56.519742+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21816530,"num_objects":5328,"num_object_clones":0,"num_object_copies":10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5329,"num_write_kb":21307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1801},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5397","reported_seq":5482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.507108+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:55:56.507108+0000","last_peered":"2024-09-18T11:55:56.507108+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:55:56.507108+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22082770,"num_objects":5393,"num_object_clones":0,"num_object_copies":10786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5395,"num_write_kb":21568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1834},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5512","reported_seq":5735,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:56.459726+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:55:56.459726+0000","last_peered":"2024-09-18T11:55:56.459726+0000","last_clean":"2024-09-18T11:55:56.459726+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:55:56.459726+0000","last_undegraded":"2024-09-18T11:55:56.459726+0000","last_fullsized":"2024-09-18T11:55:56.459726+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22568960,"num_objects":5510,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1829,"num_write_kb":7316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5379","reported_seq":8073,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:57.590644+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:55:57.590644+0000","last_peered":"2024-09-18T11:55:57.590644+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:55:57.590644+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5379,"log_dups_size":0,"ondisk_log_size":5379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22004946,"num_objects":5374,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2206,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5377,"num_write_kb":21492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":5210112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2206},{"shards":"2","objects":3167}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5407","reported_seq":5481,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.536291+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:55:56.536291+0000","last_peered":"2024-09-18T11:55:56.536291+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:55:56.536291+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5407,"log_dups_size":0,"ondisk_log_size":5407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22115538,"num_objects":5401,"num_object_clones":0,"num_object_copies":10802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5405,"num_write_kb":21600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1869},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5487","reported_seq":5539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:57.590600+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:55:57.590600+0000","last_peered":"2024-09-18T11:55:57.590600+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:55:57.590600+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5487,"log_dups_size":0,"ondisk_log_size":5487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22455506,"num_objects":5484,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":21931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":1934}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5473","reported_seq":5680,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:57.577593+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:55:57.577593+0000","last_peered":"2024-09-18T11:55:57.577593+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:55:57.577593+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5473,"log_dups_size":0,"ondisk_log_size":5473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22392832,"num_objects":5467,"num_object_clones":0,"num_object_copies":10934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1953,"num_write_kb":7812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":1874}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5382","reported_seq":5462,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.463371+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:55:56.463371+0000","last_peered":"2024-09-18T11:55:56.463371+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:55:56.463371+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22037097,"num_objects":5381,"num_object_clones":0,"num_object_copies":10762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5381,"num_write_kb":21521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1846},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5462","reported_seq":5544,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.485776+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:55:56.485776+0000","last_peered":"2024-09-18T11:55:56.485776+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:55:56.485776+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5462,"log_dups_size":0,"ondisk_log_size":5462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22341435,"num_objects":5457,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5459,"num_write_kb":21820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1935},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5382","reported_seq":5461,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.489426+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:55:56.489426+0000","last_peered":"2024-09-18T11:55:56.489426+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:55:56.489426+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5382,"log_dups_size":0,"ondisk_log_size":5382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22029522,"num_objects":5380,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5380,"num_write_kb":21514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1787},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5435","reported_seq":5516,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:55:56.528481+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:55:56.528481+0000","last_peered":"2024-09-18T11:55:56.528481+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:55:56.528481+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5435,"log_dups_size":0,"ondisk_log_size":5435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22215076,"num_objects":5427,"num_object_clones":0,"num_object_copies":10854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5432,"num_write_kb":21699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1904},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":355364916,"num_objects":86776,"num_object_clones":0,"num_object_copies":173552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2206,"num_objects_unfound":0,"num_objects_dirty":86776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70831,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1294,"num_bytes_recovered":5218921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715501568,"data_stored":715414593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86838,"ondisk_log_size":86838,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051609,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589372,"kb_used_data":287348,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93782468,"statfs":{"total":96636764160,"available":96033247232,"internally_reserved":0,"allocated":294244352,"data_stored":294031196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.317},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":7.2619999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3239999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855372,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395304,"kb_used_data":93280,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93976536,"statfs":{"total":96636764160,"available":96231972864,"internally_reserved":0,"allocated":95518720,"data_stored":95225404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,3,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614160,"kb_used_data":316360,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93757680,"statfs":{"total":96636764160,"available":96007864320,"internally_reserved":0,"allocated":323952640,"data_stored":323735862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,2,7,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6840000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":2.7639999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":4.7140000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323887104,"data_stored":323883625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96833536,"data_stored":96753519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294780928,"data_stored":294777449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:03.119 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 97 20 90052 90032 3.6246 2.04688 0.050754 0.0215457 2024-09-18T11:56:04.119 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 98 20 90590 90570 3.60906 2.10156 0.0267676 0.0216382 2024-09-18T11:56:05.119 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 99 20 91142 91122 3.59439 2.15625 0.0232965 0.0217277 2024-09-18T11:56:05.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:06.121 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:2024-09-18T11:56:06.122329+0000 min lat: 0.0021114 max lat: 2.57663 avg lat: 0.021823 2024-09-18T11:56:06.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:56:06.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 100 20 91653 91633 3.57833 1.99609 0.0297645 0.021823 2024-09-18T11:56:06.226 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:06.241 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:06.256 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-18T11:56:05.059132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":366276679,"num_objects":89442,"num_object_clones":0,"num_object_copies":178884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2059,"num_objects_unfound":0,"num_objects_dirty":89442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73497,"num_write_kb":293862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1442,"num_bytes_recovered":5821052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89505,"ondisk_log_size":89505,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1639120,"kb_used_data":720632,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281476400,"statfs":{"total":289910292480,"available":288231833600,"internally_reserved":0,"allocated":737927168,"data_stored":737203918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,8,8,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21946368,"num_objects":5358,"num_object_clones":0,"num_object_copies":10716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-280,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":21432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007247"},"pg_stats":[{"pgid":"2.f","version":"37'5665","reported_seq":5734,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.395388+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:01.395388+0000","last_peered":"2024-09-18T11:56:01.395388+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:01.395388+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23203840,"num_objects":5665,"num_object_clones":0,"num_object_copies":11330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":22660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2066},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5601","reported_seq":5756,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:02.556508+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:02.556508+0000","last_peered":"2024-09-18T11:56:02.556508+0000","last_clean":"2024-09-18T11:56:02.556508+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:02.556508+0000","last_undegraded":"2024-09-18T11:56:02.556508+0000","last_fullsized":"2024-09-18T11:56:02.556508+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22917120,"num_objects":5595,"num_object_clones":0,"num_object_copies":11190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2793,"num_write_kb":11172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5600","reported_seq":5783,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:02.607613+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:02.607613+0000","last_peered":"2024-09-18T11:56:02.607613+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:02.607613+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5600,"log_dups_size":0,"ondisk_log_size":5600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22921216,"num_objects":5596,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2458,"num_write_kb":9832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2060}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5607","reported_seq":5634,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:02.594417+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:02.594417+0000","last_peered":"2024-09-18T11:56:02.594417+0000","last_clean":"2024-09-18T11:56:02.594417+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:02.594417+0000","last_undegraded":"2024-09-18T11:56:02.594417+0000","last_fullsized":"2024-09-18T11:56:02.594417+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22966272,"num_objects":5607,"num_object_clones":0,"num_object_copies":11214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5607,"num_write_kb":22428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5535","reported_seq":5724,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:01.471558+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:01.471558+0000","last_peered":"2024-09-18T11:56:01.471558+0000","last_clean":"2024-09-18T11:56:01.471558+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:01.471558+0000","last_undegraded":"2024-09-18T11:56:01.471558+0000","last_fullsized":"2024-09-18T11:56:01.471558+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5535,"log_dups_size":0,"ondisk_log_size":5535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22642688,"num_objects":5528,"num_object_clones":0,"num_object_copies":11056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2700,"num_write_kb":10800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'5514","reported_seq":5592,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.505753+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:01.505753+0000","last_peered":"2024-09-18T11:56:01.505753+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:01.505753+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5514,"log_dups_size":0,"ondisk_log_size":5514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22566098,"num_objects":5511,"num_object_clones":0,"num_object_copies":11022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5512,"num_write_kb":22039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1984},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5586","reported_seq":5668,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.541476+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:01.541476+0000","last_peered":"2024-09-18T11:56:01.541476+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:01.541476+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22856914,"num_objects":5582,"num_object_clones":0,"num_object_copies":11164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5584,"num_write_kb":22324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2023},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5648","reported_seq":5871,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:01.508082+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:01.508082+0000","last_peered":"2024-09-18T11:56:01.508082+0000","last_clean":"2024-09-18T11:56:01.508082+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:01.508082+0000","last_undegraded":"2024-09-18T11:56:01.508082+0000","last_fullsized":"2024-09-18T11:56:01.508082+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5648,"log_dups_size":0,"ondisk_log_size":5648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23126016,"num_objects":5646,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1965,"num_write_kb":7860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5536","reported_seq":8526,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:02.555771+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:02.555771+0000","last_peered":"2024-09-18T11:56:02.555771+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:02.555771+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5536,"log_dups_size":0,"ondisk_log_size":5536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648018,"num_objects":5531,"num_object_clones":0,"num_object_copies":11062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2059,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5534,"num_write_kb":22120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1421,"num_bytes_recovered":5812224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2059},{"shards":"2","objects":3471}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5603","reported_seq":5679,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.503688+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:01.503688+0000","last_peered":"2024-09-18T11:56:01.503688+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:01.503688+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5603,"log_dups_size":0,"ondisk_log_size":5603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22918354,"num_objects":5597,"num_object_clones":0,"num_object_copies":11194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5601,"num_write_kb":22384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2065},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5629","reported_seq":5683,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:02.429449+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:02.429449+0000","last_peered":"2024-09-18T11:56:02.429449+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:02.429449+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5629,"log_dups_size":0,"ondisk_log_size":5629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23037138,"num_objects":5626,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5627,"num_write_kb":22499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2076}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5620","reported_seq":5827,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:02.530823+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:02.530823+0000","last_peered":"2024-09-18T11:56:02.530823+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:02.530823+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22994944,"num_objects":5614,"num_object_clones":0,"num_object_copies":11228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2100,"num_write_kb":8400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2021}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5542","reported_seq":5622,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.553128+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:01.553128+0000","last_peered":"2024-09-18T11:56:01.553128+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:01.553128+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5542,"log_dups_size":0,"ondisk_log_size":5542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22692457,"num_objects":5541,"num_object_clones":0,"num_object_copies":11082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5541,"num_write_kb":22161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2006},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5641","reported_seq":5723,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.518576+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:01.518576+0000","last_peered":"2024-09-18T11:56:01.518576+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:01.518576+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5641,"log_dups_size":0,"ondisk_log_size":5641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23074619,"num_objects":5636,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5638,"num_write_kb":22536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2114},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5556","reported_seq":5635,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.508104+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:01.508104+0000","last_peered":"2024-09-18T11:56:01.508104+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:01.508104+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22742226,"num_objects":5554,"num_object_clones":0,"num_object_copies":11108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":22210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1961},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5619","reported_seq":5702,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.522580+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:01.522580+0000","last_peered":"2024-09-18T11:56:01.522580+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:01.522580+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5619,"log_dups_size":0,"ondisk_log_size":5619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22968740,"num_objects":5611,"num_object_clones":0,"num_object_copies":11222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5616,"num_write_kb":22435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2088},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":366276660,"num_objects":89440,"num_object_clones":0,"num_object_copies":178880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2059,"num_objects_unfound":0,"num_objects_dirty":89440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73495,"num_write_kb":293860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":5821033,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":738025472,"data_stored":737938497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89502,"ondisk_log_size":89502,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051610,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598612,"kb_used_data":292428,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93773228,"statfs":{"total":96636764160,"available":96023785472,"internally_reserved":0,"allocated":299446272,"data_stored":299233116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":1.2969999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":3.069},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.254}]}]},{"osd":1,"up_from":35,"seq":150323855373,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413964,"kb_used_data":103620,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93957876,"statfs":{"total":96636764160,"available":96212865024,"internally_reserved":0,"allocated":106106880,"data_stored":105813564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,5,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626544,"kb_used_data":324584,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93745296,"statfs":{"total":96636764160,"available":95995183104,"internally_reserved":0,"allocated":332374016,"data_stored":332157238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.56899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":332378112,"data_stored":332374633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":105840640,"data_stored":105760623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":299806720,"data_stored":299803241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:06.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:06.958 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:06.976 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:07.002 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-18T11:56:05.059132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":366276679,"num_objects":89442,"num_object_clones":0,"num_object_copies":178884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2059,"num_objects_unfound":0,"num_objects_dirty":89442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73497,"num_write_kb":293862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1442,"num_bytes_recovered":5821052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89505,"ondisk_log_size":89505,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1639120,"kb_used_data":720632,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281476400,"statfs":{"total":289910292480,"available":288231833600,"internally_reserved":0,"allocated":737927168,"data_stored":737203918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,8,8,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21946368,"num_objects":5358,"num_object_clones":0,"num_object_copies":10716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-280,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":21432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007247"},"pg_stats":[{"pgid":"2.f","version":"37'5665","reported_seq":5734,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.395388+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:01.395388+0000","last_peered":"2024-09-18T11:56:01.395388+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:01.395388+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23203840,"num_objects":5665,"num_object_clones":0,"num_object_copies":11330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":22660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2066},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5601","reported_seq":5756,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:02.556508+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:02.556508+0000","last_peered":"2024-09-18T11:56:02.556508+0000","last_clean":"2024-09-18T11:56:02.556508+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:02.556508+0000","last_undegraded":"2024-09-18T11:56:02.556508+0000","last_fullsized":"2024-09-18T11:56:02.556508+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22917120,"num_objects":5595,"num_object_clones":0,"num_object_copies":11190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2793,"num_write_kb":11172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5600","reported_seq":5783,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:02.607613+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:02.607613+0000","last_peered":"2024-09-18T11:56:02.607613+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:02.607613+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5600,"log_dups_size":0,"ondisk_log_size":5600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22921216,"num_objects":5596,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2458,"num_write_kb":9832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2060}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5607","reported_seq":5634,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:02.594417+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:02.594417+0000","last_peered":"2024-09-18T11:56:02.594417+0000","last_clean":"2024-09-18T11:56:02.594417+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:02.594417+0000","last_undegraded":"2024-09-18T11:56:02.594417+0000","last_fullsized":"2024-09-18T11:56:02.594417+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22966272,"num_objects":5607,"num_object_clones":0,"num_object_copies":11214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5607,"num_write_kb":22428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5535","reported_seq":5724,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:01.471558+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:01.471558+0000","last_peered":"2024-09-18T11:56:01.471558+0000","last_clean":"2024-09-18T11:56:01.471558+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:01.471558+0000","last_undegraded":"2024-09-18T11:56:01.471558+0000","last_fullsized":"2024-09-18T11:56:01.471558+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5535,"log_dups_size":0,"ondisk_log_size":5535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22642688,"num_objects":5528,"num_object_clones":0,"num_object_copies":11056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2700,"num_write_kb":10800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'5514","reported_seq":5592,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.505753+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:01.505753+0000","last_peered":"2024-09-18T11:56:01.505753+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:01.505753+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5514,"log_dups_size":0,"ondisk_log_size":5514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22566098,"num_objects":5511,"num_object_clones":0,"num_object_copies":11022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5512,"num_write_kb":22039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1984},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5586","reported_seq":5668,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.541476+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:01.541476+0000","last_peered":"2024-09-18T11:56:01.541476+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:01.541476+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22856914,"num_objects":5582,"num_object_clones":0,"num_object_copies":11164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5584,"num_write_kb":22324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2023},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5648","reported_seq":5871,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:01.508082+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:01.508082+0000","last_peered":"2024-09-18T11:56:01.508082+0000","last_clean":"2024-09-18T11:56:01.508082+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:01.508082+0000","last_undegraded":"2024-09-18T11:56:01.508082+0000","last_fullsized":"2024-09-18T11:56:01.508082+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5648,"log_dups_size":0,"ondisk_log_size":5648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23126016,"num_objects":5646,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1965,"num_write_kb":7860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5536","reported_seq":8526,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:02.555771+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:02.555771+0000","last_peered":"2024-09-18T11:56:02.555771+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:02.555771+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5536,"log_dups_size":0,"ondisk_log_size":5536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648018,"num_objects":5531,"num_object_clones":0,"num_object_copies":11062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2059,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5534,"num_write_kb":22120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1421,"num_bytes_recovered":5812224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2059},{"shards":"2","objects":3471}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5603","reported_seq":5679,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.503688+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:01.503688+0000","last_peered":"2024-09-18T11:56:01.503688+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:01.503688+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5603,"log_dups_size":0,"ondisk_log_size":5603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22918354,"num_objects":5597,"num_object_clones":0,"num_object_copies":11194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5601,"num_write_kb":22384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2065},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5629","reported_seq":5683,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:02.429449+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:02.429449+0000","last_peered":"2024-09-18T11:56:02.429449+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:02.429449+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5629,"log_dups_size":0,"ondisk_log_size":5629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23037138,"num_objects":5626,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5627,"num_write_kb":22499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2076}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5620","reported_seq":5827,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:02.530823+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:02.530823+0000","last_peered":"2024-09-18T11:56:02.530823+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:02.530823+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22994944,"num_objects":5614,"num_object_clones":0,"num_object_copies":11228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2100,"num_write_kb":8400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2021}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5542","reported_seq":5622,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.553128+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:01.553128+0000","last_peered":"2024-09-18T11:56:01.553128+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:01.553128+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5542,"log_dups_size":0,"ondisk_log_size":5542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22692457,"num_objects":5541,"num_object_clones":0,"num_object_copies":11082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5541,"num_write_kb":22161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2006},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5641","reported_seq":5723,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.518576+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:01.518576+0000","last_peered":"2024-09-18T11:56:01.518576+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:01.518576+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5641,"log_dups_size":0,"ondisk_log_size":5641,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23074619,"num_objects":5636,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5638,"num_write_kb":22536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2114},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5556","reported_seq":5635,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.508104+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:01.508104+0000","last_peered":"2024-09-18T11:56:01.508104+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:01.508104+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22742226,"num_objects":5554,"num_object_clones":0,"num_object_copies":11108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":22210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1961},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5619","reported_seq":5702,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:01.522580+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:01.522580+0000","last_peered":"2024-09-18T11:56:01.522580+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:01.522580+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5619,"log_dups_size":0,"ondisk_log_size":5619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22968740,"num_objects":5611,"num_object_clones":0,"num_object_copies":11222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5616,"num_write_kb":22435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2088},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":366276660,"num_objects":89440,"num_object_clones":0,"num_object_copies":178880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2059,"num_objects_unfound":0,"num_objects_dirty":89440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73495,"num_write_kb":293860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":5821033,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":738025472,"data_stored":737938497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89502,"ondisk_log_size":89502,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051610,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598612,"kb_used_data":292428,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93773228,"statfs":{"total":96636764160,"available":96023785472,"internally_reserved":0,"allocated":299446272,"data_stored":299233116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":1.2969999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":3.069},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.254}]}]},{"osd":1,"up_from":35,"seq":150323855373,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413964,"kb_used_data":103620,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93957876,"statfs":{"total":96636764160,"available":96212865024,"internally_reserved":0,"allocated":106106880,"data_stored":105813564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,5,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626544,"kb_used_data":324584,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93745296,"statfs":{"total":96636764160,"available":95995183104,"internally_reserved":0,"allocated":332374016,"data_stored":332157238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.56899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":332378112,"data_stored":332374633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":105840640,"data_stored":105760623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":299806720,"data_stored":299803241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:07.003 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:56:07.004 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'5665", 'reported_seq': 5734, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:01.395388+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:56:01.395388+0000', 'last_peered': '2024-09-18T11:56:01.395388+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:56:01.395388+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5665, 'log_dups_size': 0, 'ondisk_log_size': 5665, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23203840, 'num_objects': 5665, 'num_object_clones': 0, 'num_object_copies': 11330, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5665, 'num_write_kb': 22660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2066}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:07.004 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:56:07.004 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'5600", 'reported_seq': 5783, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:02.607613+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:56:02.607613+0000', 'last_peered': '2024-09-18T11:56:02.607613+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:56:02.607613+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 5600, 'log_dups_size': 0, 'ondisk_log_size': 5600, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22921216, 'num_objects': 5596, 'num_object_clones': 0, 'num_object_copies': 11192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2458, 'num_write_kb': 9832, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 2060}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:07.004 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:56:07.004 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'5514", 'reported_seq': 5592, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:01.505753+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:56:01.505753+0000', 'last_peered': '2024-09-18T11:56:01.505753+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:56:01.505753+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5514, 'log_dups_size': 0, 'ondisk_log_size': 5514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22566098, 'num_objects': 5511, 'num_object_clones': 0, 'num_object_copies': 11022, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5512, 'num_write_kb': 22039, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1984}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:07.004 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:56:07.004 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'5586", 'reported_seq': 5668, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:01.541476+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:56:01.541476+0000', 'last_peered': '2024-09-18T11:56:01.541476+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:56:01.541476+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5586, 'log_dups_size': 0, 'ondisk_log_size': 5586, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22856914, 'num_objects': 5582, 'num_object_clones': 0, 'num_object_copies': 11164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5584, 'num_write_kb': 22324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2023}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:07.004 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:56:07.004 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'5536", 'reported_seq': 8526, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:02.555771+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:56:02.555771+0000', 'last_peered': '2024-09-18T11:56:02.555771+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:56:02.555771+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 5536, 'log_dups_size': 0, 'ondisk_log_size': 5536, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22648018, 'num_objects': 5531, 'num_object_clones': 0, 'num_object_copies': 11062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 2059, 'num_objects_unfound': 0, 'num_objects_dirty': 5531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5534, 'num_write_kb': 22120, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1421, 'num_bytes_recovered': 5812224, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2059}, {'shards': '2', 'objects': 3471}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:07.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:56:07.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'5603", 'reported_seq': 5679, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:01.503688+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:56:01.503688+0000', 'last_peered': '2024-09-18T11:56:01.503688+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:56:01.503688+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5603, 'log_dups_size': 0, 'ondisk_log_size': 5603, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22918354, 'num_objects': 5597, 'num_object_clones': 0, 'num_object_copies': 11194, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5601, 'num_write_kb': 22384, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2065}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:07.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:56:07.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'5629", 'reported_seq': 5683, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:02.429449+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:56:02.429449+0000', 'last_peered': '2024-09-18T11:56:02.429449+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:56:02.429449+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5629, 'log_dups_size': 0, 'ondisk_log_size': 5629, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23037138, 'num_objects': 5626, 'num_object_clones': 0, 'num_object_copies': 11252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5626, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5627, 'num_write_kb': 22499, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 2076}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:07.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:56:07.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'5620", 'reported_seq': 5827, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:02.530823+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:56:02.530823+0000', 'last_peered': '2024-09-18T11:56:02.530823+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:56:02.530823+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 5620, 'log_dups_size': 0, 'ondisk_log_size': 5620, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22994944, 'num_objects': 5614, 'num_object_clones': 0, 'num_object_copies': 11228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2100, 'num_write_kb': 8400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 2021}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:07.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:56:07.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'5542", 'reported_seq': 5622, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:01.553128+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:56:01.553128+0000', 'last_peered': '2024-09-18T11:56:01.553128+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:56:01.553128+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5542, 'log_dups_size': 0, 'ondisk_log_size': 5542, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22692457, 'num_objects': 5541, 'num_object_clones': 0, 'num_object_copies': 11082, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5541, 'num_write_kb': 22161, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2006}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:07.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:56:07.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'5641", 'reported_seq': 5723, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:01.518576+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:56:01.518576+0000', 'last_peered': '2024-09-18T11:56:01.518576+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:56:01.518576+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5641, 'log_dups_size': 0, 'ondisk_log_size': 5641, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23074619, 'num_objects': 5636, 'num_object_clones': 0, 'num_object_copies': 11272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5638, 'num_write_kb': 22536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2114}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:07.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:56:07.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'5556", 'reported_seq': 5635, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:01.508104+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:56:01.508104+0000', 'last_peered': '2024-09-18T11:56:01.508104+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:56:01.508104+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5556, 'log_dups_size': 0, 'ondisk_log_size': 5556, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22742226, 'num_objects': 5554, 'num_object_clones': 0, 'num_object_copies': 11108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5554, 'num_write_kb': 22210, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1961}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:07.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:56:07.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'5619", 'reported_seq': 5702, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:01.522580+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:56:01.522580+0000', 'last_peered': '2024-09-18T11:56:01.522580+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:56:01.522580+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5619, 'log_dups_size': 0, 'ondisk_log_size': 5619, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22968740, 'num_objects': 5611, 'num_object_clones': 0, 'num_object_copies': 11222, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5611, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5616, 'num_write_kb': 22435, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2088}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:07.006 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:07.121 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 101 20 92134 92114 3.56151 1.87891 0.0166587 0.0219282 2024-09-18T11:56:07.709 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:07.709 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/182186 objects degraded (21.456%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":167,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":91093,"data_bytes":373039175,"bytes_used":1695240192,"bytes_avail":288215052288,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":182186,"degraded_ratio":0.21456094321188235,"misplaced_objects":2059,"misplaced_total":182186,"misplaced_ratio":0.011301636788776305,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":95515,"recovering_keys_per_sec":0,"num_objects_recovered":280,"num_bytes_recovered":1146880,"num_keys_recovered":0,"write_bytes_sec":2390960,"read_op_per_sec":0,"write_op_per_sec":583},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:07.741 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:07.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:08.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 102 20 92632 92612 3.54567 1.94531 0.0370914 0.0220238 2024-09-18T11:56:08.505 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:08.514 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:08.524 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-09-18T11:56:07.059606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":373039175,"num_objects":91093,"num_object_clones":0,"num_object_copies":182186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2059,"num_objects_unfound":0,"num_objects_dirty":91093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75148,"num_write_kb":300466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1442,"num_bytes_recovered":5821052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91156,"ondisk_log_size":91156,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1655508,"kb_used_data":737020,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281460012,"statfs":{"total":289910292480,"available":288215052288,"internally_reserved":0,"allocated":754708480,"data_stored":753985230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,4,8,12,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28708864,"num_objects":7009,"num_object_clones":0,"num_object_copies":14018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-280,"num_objects_unfound":0,"num_objects_dirty":7009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7009,"num_write_kb":28036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007250"},"pg_stats":[{"pgid":"2.f","version":"37'5804","reported_seq":5873,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.505811+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:06.505811+0000","last_peered":"2024-09-18T11:56:06.505811+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:06.505811+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5804,"log_dups_size":0,"ondisk_log_size":5804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23773184,"num_objects":5804,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5804,"num_write_kb":23216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2205},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5601","reported_seq":5756,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:02.556508+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:02.556508+0000","last_peered":"2024-09-18T11:56:02.556508+0000","last_clean":"2024-09-18T11:56:02.556508+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:02.556508+0000","last_undegraded":"2024-09-18T11:56:02.556508+0000","last_fullsized":"2024-09-18T11:56:02.556508+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22917120,"num_objects":5595,"num_object_clones":0,"num_object_copies":11190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2793,"num_write_kb":11172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5600","reported_seq":5783,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:02.607613+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:02.607613+0000","last_peered":"2024-09-18T11:56:02.607613+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:02.607613+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5600,"log_dups_size":0,"ondisk_log_size":5600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22921216,"num_objects":5596,"num_object_clones":0,"num_object_copies":11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2458,"num_write_kb":9832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2060}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5607","reported_seq":5634,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:02.594417+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:02.594417+0000","last_peered":"2024-09-18T11:56:02.594417+0000","last_clean":"2024-09-18T11:56:02.594417+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:02.594417+0000","last_undegraded":"2024-09-18T11:56:02.594417+0000","last_fullsized":"2024-09-18T11:56:02.594417+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22966272,"num_objects":5607,"num_object_clones":0,"num_object_copies":11214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5607,"num_write_kb":22428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5710","reported_seq":5899,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:06.519384+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:06.519384+0000","last_peered":"2024-09-18T11:56:06.519384+0000","last_clean":"2024-09-18T11:56:06.519384+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:06.519384+0000","last_undegraded":"2024-09-18T11:56:06.519384+0000","last_fullsized":"2024-09-18T11:56:06.519384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5710,"log_dups_size":0,"ondisk_log_size":5710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23359488,"num_objects":5703,"num_object_clones":0,"num_object_copies":11406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2875,"num_write_kb":11500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'5694","reported_seq":5772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.547558+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:06.547558+0000","last_peered":"2024-09-18T11:56:06.547558+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:06.547558+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5694,"log_dups_size":0,"ondisk_log_size":5694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23303378,"num_objects":5691,"num_object_clones":0,"num_object_copies":11382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5692,"num_write_kb":22759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2164},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5760","reported_seq":5844,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.555503+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:06.555503+0000","last_peered":"2024-09-18T11:56:06.555503+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:06.555503+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5760,"log_dups_size":0,"ondisk_log_size":5760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23569618,"num_objects":5756,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5758,"num_write_kb":23020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2197},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5795","reported_seq":6018,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:06.570903+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:06.570903+0000","last_peered":"2024-09-18T11:56:06.570903+0000","last_clean":"2024-09-18T11:56:06.570903+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:06.570903+0000","last_undegraded":"2024-09-18T11:56:06.570903+0000","last_fullsized":"2024-09-18T11:56:06.570903+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5795,"log_dups_size":0,"ondisk_log_size":5795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23728128,"num_objects":5793,"num_object_clones":0,"num_object_copies":11586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2112,"num_write_kb":8448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5536","reported_seq":8526,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:02.555771+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:02.555771+0000","last_peered":"2024-09-18T11:56:02.555771+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:02.555771+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5536,"log_dups_size":0,"ondisk_log_size":5536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648018,"num_objects":5531,"num_object_clones":0,"num_object_copies":11062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":2059,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5534,"num_write_kb":22120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1421,"num_bytes_recovered":5812224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2059},{"shards":"2","objects":3471}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5773","reported_seq":5846,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.519932+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:06.519932+0000","last_peered":"2024-09-18T11:56:06.519932+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:06.519932+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23614674,"num_objects":5767,"num_object_clones":0,"num_object_copies":11534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5771,"num_write_kb":23064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2235},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5629","reported_seq":5683,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:02.429449+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:02.429449+0000","last_peered":"2024-09-18T11:56:02.429449+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:02.429449+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5629,"log_dups_size":0,"ondisk_log_size":5629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23037138,"num_objects":5626,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5627,"num_write_kb":22499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2076}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5620","reported_seq":5827,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:02.530823+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:02.530823+0000","last_peered":"2024-09-18T11:56:02.530823+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:02.530823+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22994944,"num_objects":5614,"num_object_clones":0,"num_object_copies":11228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2100,"num_write_kb":8400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2021}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5681","reported_seq":5761,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.534875+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:06.534875+0000","last_peered":"2024-09-18T11:56:06.534875+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:06.534875+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23261801,"num_objects":5680,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5680,"num_write_kb":22717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2145},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5818","reported_seq":5900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.514028+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:06.514028+0000","last_peered":"2024-09-18T11:56:06.514028+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:06.514028+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23799611,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5815,"num_write_kb":23244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2291},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5722","reported_seq":5801,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.568201+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:06.568201+0000","last_peered":"2024-09-18T11:56:06.568201+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:06.568201+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23422162,"num_objects":5720,"num_object_clones":0,"num_object_copies":11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5720,"num_write_kb":22874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2127},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5803","reported_seq":5886,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.483090+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:06.483090+0000","last_peered":"2024-09-18T11:56:06.483090+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:06.483090+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23722404,"num_objects":5795,"num_object_clones":0,"num_object_copies":11590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5800,"num_write_kb":23171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2272},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":373039156,"num_objects":91091,"num_object_clones":0,"num_object_copies":182182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":2059,"num_objects_unfound":0,"num_objects_dirty":91091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75146,"num_write_kb":300464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":5821033,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":754802688,"data_stored":754715713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91153,"ondisk_log_size":91153,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051610,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598612,"kb_used_data":292428,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93773228,"statfs":{"total":96636764160,"available":96023785472,"internally_reserved":0,"allocated":299446272,"data_stored":299233116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":1.2969999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":3.069},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.254}]}]},{"osd":1,"up_from":35,"seq":150323855374,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414208,"kb_used_data":112184,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93957632,"statfs":{"total":96636764160,"available":96212615168,"internally_reserved":0,"allocated":114876416,"data_stored":114583100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.39100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642688,"kb_used_data":332408,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93729152,"statfs":{"total":96636764160,"available":95978651648,"internally_reserved":0,"allocated":340385792,"data_stored":340169014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.251}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":340393984,"data_stored":340390505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114601984,"data_stored":114521967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":299806720,"data_stored":299803241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:09.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 103 20 93150 93130 3.53089 2.02344 0.0350274 0.0221166 2024-09-18T11:56:10.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 104 20 93682 93662 3.51692 2.07812 0.11114 0.0222052 2024-09-18T11:56:11.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 105 20 94212 94192 3.50314 2.07031 0.0612482 0.0222922 2024-09-18T11:56:11.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:12.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 106 20 94709 94689 3.48841 1.94141 0.0121567 0.0223825 2024-09-18T11:56:12.164 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:12.168 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:12.188 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-18T11:56:11.060602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":377102407,"num_objects":92085,"num_object_clones":0,"num_object_copies":184170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1913,"num_objects_unfound":0,"num_objects_dirty":92085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76140,"num_write_kb":304434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1588,"num_bytes_recovered":6419068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92148,"ondisk_log_size":92148,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1664684,"kb_used_data":742036,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281450836,"statfs":{"total":289910292480,"available":288205656064,"internally_reserved":0,"allocated":759844864,"data_stored":759121614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,4,8,12,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21737472,"num_objects":5307,"num_object_clones":0,"num_object_copies":10614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-293,"num_objects_unfound":0,"num_objects_dirty":5307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5307,"num_write_kb":21228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":1200128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007406"},"pg_stats":[{"pgid":"2.f","version":"37'5804","reported_seq":5873,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.505811+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:06.505811+0000","last_peered":"2024-09-18T11:56:06.505811+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:06.505811+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5804,"log_dups_size":0,"ondisk_log_size":5804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23773184,"num_objects":5804,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5804,"num_write_kb":23216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2205},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5762","reported_seq":5918,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.585223+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:07.585223+0000","last_peered":"2024-09-18T11:56:07.585223+0000","last_clean":"2024-09-18T11:56:07.585223+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:07.585223+0000","last_undegraded":"2024-09-18T11:56:07.585223+0000","last_fullsized":"2024-09-18T11:56:07.585223+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23576576,"num_objects":5756,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2954,"num_write_kb":11816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5776","reported_seq":5957,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:07.562664+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:07.562664+0000","last_peered":"2024-09-18T11:56:07.562664+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:07.562664+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23642112,"num_objects":5772,"num_object_clones":0,"num_object_copies":11544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2634,"num_write_kb":10536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2236}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5749","reported_seq":5776,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.585009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:07.585009+0000","last_peered":"2024-09-18T11:56:07.585009+0000","last_clean":"2024-09-18T11:56:07.585009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:07.585009+0000","last_undegraded":"2024-09-18T11:56:07.585009+0000","last_fullsized":"2024-09-18T11:56:07.585009+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5749,"log_dups_size":0,"ondisk_log_size":5749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23547904,"num_objects":5749,"num_object_clones":0,"num_object_copies":11498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5749,"num_write_kb":22996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5710","reported_seq":5899,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:06.519384+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:06.519384+0000","last_peered":"2024-09-18T11:56:06.519384+0000","last_clean":"2024-09-18T11:56:06.519384+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:06.519384+0000","last_undegraded":"2024-09-18T11:56:06.519384+0000","last_fullsized":"2024-09-18T11:56:06.519384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5710,"log_dups_size":0,"ondisk_log_size":5710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23359488,"num_objects":5703,"num_object_clones":0,"num_object_copies":11406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2875,"num_write_kb":11500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'5694","reported_seq":5772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.547558+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:06.547558+0000","last_peered":"2024-09-18T11:56:06.547558+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:06.547558+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5694,"log_dups_size":0,"ondisk_log_size":5694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23303378,"num_objects":5691,"num_object_clones":0,"num_object_copies":11382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5692,"num_write_kb":22759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2164},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5760","reported_seq":5844,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.555503+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:06.555503+0000","last_peered":"2024-09-18T11:56:06.555503+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:06.555503+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5760,"log_dups_size":0,"ondisk_log_size":5760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23569618,"num_objects":5756,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5758,"num_write_kb":23020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2197},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5795","reported_seq":6018,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:06.570903+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:06.570903+0000","last_peered":"2024-09-18T11:56:06.570903+0000","last_clean":"2024-09-18T11:56:06.570903+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:06.570903+0000","last_undegraded":"2024-09-18T11:56:06.570903+0000","last_fullsized":"2024-09-18T11:56:06.570903+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5795,"log_dups_size":0,"ondisk_log_size":5795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23728128,"num_objects":5793,"num_object_clones":0,"num_object_copies":11586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2112,"num_write_kb":8448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5718","reported_seq":9000,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:07.601155+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:07.601155+0000","last_peered":"2024-09-18T11:56:07.601155+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:07.601155+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23393490,"num_objects":5713,"num_object_clones":0,"num_object_copies":11426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1913,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":22848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":6410240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1913},{"shards":"2","objects":3799}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5773","reported_seq":5846,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.519932+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:06.519932+0000","last_peered":"2024-09-18T11:56:06.519932+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:06.519932+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23614674,"num_objects":5767,"num_object_clones":0,"num_object_copies":11534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5771,"num_write_kb":23064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2235},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5804","reported_seq":5858,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:07.525884+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:07.525884+0000","last_peered":"2024-09-18T11:56:07.525884+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:07.525884+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5804,"log_dups_size":0,"ondisk_log_size":5804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23753938,"num_objects":5801,"num_object_clones":0,"num_object_copies":11602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5802,"num_write_kb":23199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2251}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5776","reported_seq":5983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:07.556254+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:07.556254+0000","last_peered":"2024-09-18T11:56:07.556254+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:07.556254+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23633920,"num_objects":5770,"num_object_clones":0,"num_object_copies":11540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2256,"num_write_kb":9024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2177}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5681","reported_seq":5761,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.534875+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:06.534875+0000","last_peered":"2024-09-18T11:56:06.534875+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:06.534875+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23261801,"num_objects":5680,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5680,"num_write_kb":22717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2145},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5818","reported_seq":5900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.514028+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:06.514028+0000","last_peered":"2024-09-18T11:56:06.514028+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:06.514028+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23799611,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5815,"num_write_kb":23244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2291},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5722","reported_seq":5801,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.568201+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:06.568201+0000","last_peered":"2024-09-18T11:56:06.568201+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:06.568201+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23422162,"num_objects":5720,"num_object_clones":0,"num_object_copies":11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5720,"num_write_kb":22874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2127},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5803","reported_seq":5886,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.483090+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:06.483090+0000","last_peered":"2024-09-18T11:56:06.483090+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:06.483090+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23722404,"num_objects":5795,"num_object_clones":0,"num_object_copies":11590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5800,"num_write_kb":23171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2272},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":377102388,"num_objects":92083,"num_object_clones":0,"num_object_copies":184166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1913,"num_objects_unfound":0,"num_objects_dirty":92083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76138,"num_write_kb":304432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1587,"num_bytes_recovered":6419049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":760201216,"data_stored":760114241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92145,"ondisk_log_size":92145,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051611,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607788,"kb_used_data":297444,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93764052,"statfs":{"total":96636764160,"available":96014389248,"internally_reserved":0,"allocated":304582656,"data_stored":304369500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":1.2969999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":3.069},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.254}]}]},{"osd":1,"up_from":35,"seq":150323855374,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414208,"kb_used_data":112184,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93957632,"statfs":{"total":96636764160,"available":96212615168,"internally_reserved":0,"allocated":114876416,"data_stored":114583100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.39100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642688,"kb_used_data":332408,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93729152,"statfs":{"total":96636764160,"available":95978651648,"internally_reserved":0,"allocated":340385792,"data_stored":340169014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.251}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":340393984,"data_stored":340390505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114601984,"data_stored":114521967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":305205248,"data_stored":305201769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:12.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:12.873 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:12.878 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:12.900 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-18T11:56:11.060602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":377102407,"num_objects":92085,"num_object_clones":0,"num_object_copies":184170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1913,"num_objects_unfound":0,"num_objects_dirty":92085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76140,"num_write_kb":304434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1588,"num_bytes_recovered":6419068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92148,"ondisk_log_size":92148,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1664684,"kb_used_data":742036,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281450836,"statfs":{"total":289910292480,"available":288205656064,"internally_reserved":0,"allocated":759844864,"data_stored":759121614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,4,8,12,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21737472,"num_objects":5307,"num_object_clones":0,"num_object_copies":10614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-293,"num_objects_unfound":0,"num_objects_dirty":5307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5307,"num_write_kb":21228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":1200128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007406"},"pg_stats":[{"pgid":"2.f","version":"37'5804","reported_seq":5873,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.505811+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:06.505811+0000","last_peered":"2024-09-18T11:56:06.505811+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:06.505811+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5804,"log_dups_size":0,"ondisk_log_size":5804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23773184,"num_objects":5804,"num_object_clones":0,"num_object_copies":11608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5804,"num_write_kb":23216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2205},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5762","reported_seq":5918,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.585223+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:07.585223+0000","last_peered":"2024-09-18T11:56:07.585223+0000","last_clean":"2024-09-18T11:56:07.585223+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:07.585223+0000","last_undegraded":"2024-09-18T11:56:07.585223+0000","last_fullsized":"2024-09-18T11:56:07.585223+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23576576,"num_objects":5756,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2954,"num_write_kb":11816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5776","reported_seq":5957,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:07.562664+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:07.562664+0000","last_peered":"2024-09-18T11:56:07.562664+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:07.562664+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23642112,"num_objects":5772,"num_object_clones":0,"num_object_copies":11544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2634,"num_write_kb":10536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2236}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5749","reported_seq":5776,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:07.585009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:07.585009+0000","last_peered":"2024-09-18T11:56:07.585009+0000","last_clean":"2024-09-18T11:56:07.585009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:07.585009+0000","last_undegraded":"2024-09-18T11:56:07.585009+0000","last_fullsized":"2024-09-18T11:56:07.585009+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5749,"log_dups_size":0,"ondisk_log_size":5749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23547904,"num_objects":5749,"num_object_clones":0,"num_object_copies":11498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5749,"num_write_kb":22996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5710","reported_seq":5899,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:06.519384+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:06.519384+0000","last_peered":"2024-09-18T11:56:06.519384+0000","last_clean":"2024-09-18T11:56:06.519384+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:06.519384+0000","last_undegraded":"2024-09-18T11:56:06.519384+0000","last_fullsized":"2024-09-18T11:56:06.519384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5710,"log_dups_size":0,"ondisk_log_size":5710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23359488,"num_objects":5703,"num_object_clones":0,"num_object_copies":11406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2875,"num_write_kb":11500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'5694","reported_seq":5772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.547558+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:06.547558+0000","last_peered":"2024-09-18T11:56:06.547558+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:06.547558+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5694,"log_dups_size":0,"ondisk_log_size":5694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23303378,"num_objects":5691,"num_object_clones":0,"num_object_copies":11382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5692,"num_write_kb":22759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2164},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5760","reported_seq":5844,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.555503+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:06.555503+0000","last_peered":"2024-09-18T11:56:06.555503+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:06.555503+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5760,"log_dups_size":0,"ondisk_log_size":5760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23569618,"num_objects":5756,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5758,"num_write_kb":23020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2197},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5795","reported_seq":6018,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:06.570903+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:06.570903+0000","last_peered":"2024-09-18T11:56:06.570903+0000","last_clean":"2024-09-18T11:56:06.570903+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:06.570903+0000","last_undegraded":"2024-09-18T11:56:06.570903+0000","last_fullsized":"2024-09-18T11:56:06.570903+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5795,"log_dups_size":0,"ondisk_log_size":5795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23728128,"num_objects":5793,"num_object_clones":0,"num_object_copies":11586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2112,"num_write_kb":8448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5718","reported_seq":9000,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:07.601155+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:07.601155+0000","last_peered":"2024-09-18T11:56:07.601155+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:07.601155+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23393490,"num_objects":5713,"num_object_clones":0,"num_object_copies":11426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1913,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":22848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":6410240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1913},{"shards":"2","objects":3799}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5773","reported_seq":5846,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.519932+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:06.519932+0000","last_peered":"2024-09-18T11:56:06.519932+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:06.519932+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5773,"log_dups_size":0,"ondisk_log_size":5773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23614674,"num_objects":5767,"num_object_clones":0,"num_object_copies":11534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5771,"num_write_kb":23064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2235},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5804","reported_seq":5858,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:07.525884+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:07.525884+0000","last_peered":"2024-09-18T11:56:07.525884+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:07.525884+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5804,"log_dups_size":0,"ondisk_log_size":5804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23753938,"num_objects":5801,"num_object_clones":0,"num_object_copies":11602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5802,"num_write_kb":23199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2251}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5776","reported_seq":5983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:07.556254+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:07.556254+0000","last_peered":"2024-09-18T11:56:07.556254+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:07.556254+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23633920,"num_objects":5770,"num_object_clones":0,"num_object_copies":11540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2256,"num_write_kb":9024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2177}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5681","reported_seq":5761,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.534875+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:06.534875+0000","last_peered":"2024-09-18T11:56:06.534875+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:06.534875+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23261801,"num_objects":5680,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5680,"num_write_kb":22717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2145},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5818","reported_seq":5900,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.514028+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:06.514028+0000","last_peered":"2024-09-18T11:56:06.514028+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:06.514028+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23799611,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5815,"num_write_kb":23244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2291},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5722","reported_seq":5801,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.568201+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:06.568201+0000","last_peered":"2024-09-18T11:56:06.568201+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:06.568201+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23422162,"num_objects":5720,"num_object_clones":0,"num_object_copies":11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5720,"num_write_kb":22874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2127},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5803","reported_seq":5886,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:06.483090+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:06.483090+0000","last_peered":"2024-09-18T11:56:06.483090+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:06.483090+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23722404,"num_objects":5795,"num_object_clones":0,"num_object_copies":11590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5800,"num_write_kb":23171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2272},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":377102388,"num_objects":92083,"num_object_clones":0,"num_object_copies":184166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1913,"num_objects_unfound":0,"num_objects_dirty":92083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76138,"num_write_kb":304432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1587,"num_bytes_recovered":6419049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":760201216,"data_stored":760114241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92145,"ondisk_log_size":92145,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051611,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607788,"kb_used_data":297444,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93764052,"statfs":{"total":96636764160,"available":96014389248,"internally_reserved":0,"allocated":304582656,"data_stored":304369500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":1.2969999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":3.069},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.254}]}]},{"osd":1,"up_from":35,"seq":150323855374,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414208,"kb_used_data":112184,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93957632,"statfs":{"total":96636764160,"available":96212615168,"internally_reserved":0,"allocated":114876416,"data_stored":114583100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.39100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642688,"kb_used_data":332408,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93729152,"statfs":{"total":96636764160,"available":95978651648,"internally_reserved":0,"allocated":340385792,"data_stored":340169014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.251}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.27400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":340393984,"data_stored":340390505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114601984,"data_stored":114521967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":305205248,"data_stored":305201769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:12.901 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:56:12.901 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'5804", 'reported_seq': 5873, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:06.505811+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:56:06.505811+0000', 'last_peered': '2024-09-18T11:56:06.505811+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:56:06.505811+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5804, 'log_dups_size': 0, 'ondisk_log_size': 5804, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23773184, 'num_objects': 5804, 'num_object_clones': 0, 'num_object_copies': 11608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5804, 'num_write_kb': 23216, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2205}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:12.901 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:56:12.901 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'5776", 'reported_seq': 5957, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:07.562664+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:56:07.562664+0000', 'last_peered': '2024-09-18T11:56:07.562664+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:56:07.562664+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 5776, 'log_dups_size': 0, 'ondisk_log_size': 5776, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23642112, 'num_objects': 5772, 'num_object_clones': 0, 'num_object_copies': 11544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2634, 'num_write_kb': 10536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 2236}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:12.902 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:56:12.902 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'5694", 'reported_seq': 5772, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:06.547558+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:56:06.547558+0000', 'last_peered': '2024-09-18T11:56:06.547558+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:56:06.547558+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5694, 'log_dups_size': 0, 'ondisk_log_size': 5694, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23303378, 'num_objects': 5691, 'num_object_clones': 0, 'num_object_copies': 11382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5692, 'num_write_kb': 22759, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2164}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:12.902 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:56:12.902 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'5760", 'reported_seq': 5844, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:06.555503+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:56:06.555503+0000', 'last_peered': '2024-09-18T11:56:06.555503+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:56:06.555503+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5760, 'log_dups_size': 0, 'ondisk_log_size': 5760, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23569618, 'num_objects': 5756, 'num_object_clones': 0, 'num_object_copies': 11512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5758, 'num_write_kb': 23020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2197}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:12.902 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:56:12.902 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'5718", 'reported_seq': 9000, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:07.601155+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:56:07.601155+0000', 'last_peered': '2024-09-18T11:56:07.601155+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:56:07.601155+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 5718, 'log_dups_size': 0, 'ondisk_log_size': 5718, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23393490, 'num_objects': 5713, 'num_object_clones': 0, 'num_object_copies': 11426, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 1913, 'num_objects_unfound': 0, 'num_objects_dirty': 5713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5716, 'num_write_kb': 22848, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1567, 'num_bytes_recovered': 6410240, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1913}, {'shards': '2', 'objects': 3799}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:12.902 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:56:12.902 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'5773", 'reported_seq': 5846, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:06.519932+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:56:06.519932+0000', 'last_peered': '2024-09-18T11:56:06.519932+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:56:06.519932+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5773, 'log_dups_size': 0, 'ondisk_log_size': 5773, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23614674, 'num_objects': 5767, 'num_object_clones': 0, 'num_object_copies': 11534, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5767, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5771, 'num_write_kb': 23064, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2235}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:12.902 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:56:12.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'5804", 'reported_seq': 5858, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:07.525884+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:56:07.525884+0000', 'last_peered': '2024-09-18T11:56:07.525884+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:56:07.525884+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5804, 'log_dups_size': 0, 'ondisk_log_size': 5804, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23753938, 'num_objects': 5801, 'num_object_clones': 0, 'num_object_copies': 11602, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5802, 'num_write_kb': 23199, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 2251}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:12.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:56:12.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'5776", 'reported_seq': 5983, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:07.556254+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:56:07.556254+0000', 'last_peered': '2024-09-18T11:56:07.556254+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:56:07.556254+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 5776, 'log_dups_size': 0, 'ondisk_log_size': 5776, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23633920, 'num_objects': 5770, 'num_object_clones': 0, 'num_object_copies': 11540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2256, 'num_write_kb': 9024, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 2177}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:12.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:56:12.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'5681", 'reported_seq': 5761, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:06.534875+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:56:06.534875+0000', 'last_peered': '2024-09-18T11:56:06.534875+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:56:06.534875+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5681, 'log_dups_size': 0, 'ondisk_log_size': 5681, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23261801, 'num_objects': 5680, 'num_object_clones': 0, 'num_object_copies': 11360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5680, 'num_write_kb': 22717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2145}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:12.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:56:12.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'5818", 'reported_seq': 5900, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:06.514028+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:56:06.514028+0000', 'last_peered': '2024-09-18T11:56:06.514028+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:56:06.514028+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5818, 'log_dups_size': 0, 'ondisk_log_size': 5818, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23799611, 'num_objects': 5813, 'num_object_clones': 0, 'num_object_copies': 11626, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5815, 'num_write_kb': 23244, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2291}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:12.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:56:12.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'5722", 'reported_seq': 5801, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:06.568201+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:56:06.568201+0000', 'last_peered': '2024-09-18T11:56:06.568201+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:56:06.568201+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5722, 'log_dups_size': 0, 'ondisk_log_size': 5722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23422162, 'num_objects': 5720, 'num_object_clones': 0, 'num_object_copies': 11440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5720, 'num_write_kb': 22874, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2127}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:12.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:56:12.904 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'5803", 'reported_seq': 5886, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:06.483090+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:56:06.483090+0000', 'last_peered': '2024-09-18T11:56:06.483090+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:56:06.483090+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5803, 'log_dups_size': 0, 'ondisk_log_size': 5803, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23722404, 'num_objects': 5795, 'num_object_clones': 0, 'num_object_copies': 11590, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5795, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5800, 'num_write_kb': 23171, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2272}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:12.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:13.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 107 20 95203 95183 3.47384 1.92969 0.0473787 0.022481 2024-09-18T11:56:13.612 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:13.612 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/189294 objects degraded (20.650%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":173,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":94647,"data_bytes":387596359,"bytes_used":1704960000,"bytes_avail":288205332480,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":189294,"degraded_ratio":0.20650416811943326,"misplaced_objects":1755,"misplaced_total":189294,"misplaced_ratio":0.0092712922755079398,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":153803,"recovering_keys_per_sec":0,"num_objects_recovered":451,"num_bytes_recovered":1847296,"num_keys_recovered":0,"write_bytes_sec":2683552,"read_op_per_sec":0,"write_op_per_sec":655},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:13.656 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:13.656 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:14.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 108 20 95694 95674 3.45944 1.91797 0.0263824 0.0225734 2024-09-18T11:56:14.398 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:14.409 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:14.431 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-18T11:56:13.064352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":387596359,"num_objects":94647,"num_object_clones":0,"num_object_copies":189294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1755,"num_objects_unfound":0,"num_objects_dirty":94647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78702,"num_write_kb":314682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1746,"num_bytes_recovered":7066236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94710,"ondisk_log_size":94710,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1665000,"kb_used_data":763148,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281450520,"statfs":{"total":289910292480,"available":288205332480,"internally_reserved":0,"allocated":781463552,"data_stored":780740302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,2,8,6,8,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32231424,"num_objects":7869,"num_object_clones":0,"num_object_copies":15738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-451,"num_objects_unfound":0,"num_objects_dirty":7869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":31476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":1847296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010730"},"pg_stats":[{"pgid":"2.f","version":"37'5966","reported_seq":6035,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:11.555959+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:11.555959+0000","last_peered":"2024-09-18T11:56:11.555959+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:11.555959+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24436736,"num_objects":5966,"num_object_clones":0,"num_object_copies":11932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5966,"num_write_kb":23864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2367},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5897","reported_seq":6053,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:12.605239+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:12.605239+0000","last_peered":"2024-09-18T11:56:12.605239+0000","last_clean":"2024-09-18T11:56:12.605239+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:12.605239+0000","last_undegraded":"2024-09-18T11:56:12.605239+0000","last_fullsized":"2024-09-18T11:56:12.605239+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5897,"log_dups_size":0,"ondisk_log_size":5897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24129536,"num_objects":5891,"num_object_clones":0,"num_object_copies":11782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3089,"num_write_kb":12356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5941","reported_seq":6124,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:12.593225+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:12.593225+0000","last_peered":"2024-09-18T11:56:12.593225+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:12.593225+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24317952,"num_objects":5937,"num_object_clones":0,"num_object_copies":11874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2799,"num_write_kb":11196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2401}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5903","reported_seq":5930,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:12.579952+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:12.579952+0000","last_peered":"2024-09-18T11:56:12.579952+0000","last_clean":"2024-09-18T11:56:12.579952+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:12.579952+0000","last_undegraded":"2024-09-18T11:56:12.579952+0000","last_fullsized":"2024-09-18T11:56:12.579952+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5903,"log_dups_size":0,"ondisk_log_size":5903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24178688,"num_objects":5903,"num_object_clones":0,"num_object_copies":11806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5903,"num_write_kb":23612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'5867","reported_seq":6056,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:11.530183+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:11.530183+0000","last_peered":"2024-09-18T11:56:11.530183+0000","last_clean":"2024-09-18T11:56:11.530183+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:11.530183+0000","last_undegraded":"2024-09-18T11:56:11.530183+0000","last_fullsized":"2024-09-18T11:56:11.530183+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":5867,"log_dups_size":0,"ondisk_log_size":5867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24002560,"num_objects":5860,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3032,"num_write_kb":12128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'5854","reported_seq":5933,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:11.500817+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:11.500817+0000","last_peered":"2024-09-18T11:56:11.500817+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:11.500817+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23958738,"num_objects":5851,"num_object_clones":0,"num_object_copies":11702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":23399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2324},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'5922","reported_seq":6007,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:11.404163+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:11.404163+0000","last_peered":"2024-09-18T11:56:11.404163+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:11.404163+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5922,"log_dups_size":0,"ondisk_log_size":5922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24233170,"num_objects":5918,"num_object_clones":0,"num_object_copies":11836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5920,"num_write_kb":23668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2359},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'5929","reported_seq":6152,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:11.545567+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:11.545567+0000","last_peered":"2024-09-18T11:56:11.545567+0000","last_clean":"2024-09-18T11:56:11.545567+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:11.545567+0000","last_undegraded":"2024-09-18T11:56:11.545567+0000","last_fullsized":"2024-09-18T11:56:11.545567+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":5929,"log_dups_size":0,"ondisk_log_size":5929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24276992,"num_objects":5927,"num_object_clones":0,"num_object_copies":11854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2246,"num_write_kb":8984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5888","reported_seq":9485,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:12.614050+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:12.614050+0000","last_peered":"2024-09-18T11:56:12.614050+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:12.614050+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5888,"log_dups_size":0,"ondisk_log_size":5888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24089810,"num_objects":5883,"num_object_clones":0,"num_object_copies":11766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1755,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5886,"num_write_kb":23528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1725,"num_bytes_recovered":7057408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1755},{"shards":"2","objects":4127}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'5928","reported_seq":6003,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:11.543089+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:11.543089+0000","last_peered":"2024-09-18T11:56:11.543089+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:11.543089+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24249554,"num_objects":5922,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":23684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2390},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5989","reported_seq":6043,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:12.502608+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:12.502608+0000","last_peered":"2024-09-18T11:56:12.502608+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:12.502608+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5989,"log_dups_size":0,"ondisk_log_size":5989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24511698,"num_objects":5986,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5987,"num_write_kb":23939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2436}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5946","reported_seq":6153,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:12.552845+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:12.552845+0000","last_peered":"2024-09-18T11:56:12.552845+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:12.552845+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24330240,"num_objects":5940,"num_object_clones":0,"num_object_copies":11880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2426,"num_write_kb":9704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5842","reported_seq":5922,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:11.495518+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:11.495518+0000","last_peered":"2024-09-18T11:56:11.495518+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:11.495518+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5842,"log_dups_size":0,"ondisk_log_size":5842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23921257,"num_objects":5841,"num_object_clones":0,"num_object_copies":11682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5841,"num_write_kb":23361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2306},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'5984","reported_seq":6066,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:11.529240+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:11.529240+0000","last_peered":"2024-09-18T11:56:11.529240+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:11.529240+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5984,"log_dups_size":0,"ondisk_log_size":5984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24479547,"num_objects":5979,"num_object_clones":0,"num_object_copies":11958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5981,"num_write_kb":23908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2457},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'5883","reported_seq":5960,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:11.488600+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:11.488600+0000","last_peered":"2024-09-18T11:56:11.488600+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:11.488600+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5883,"log_dups_size":0,"ondisk_log_size":5883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24081618,"num_objects":5881,"num_object_clones":0,"num_object_copies":11762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5881,"num_write_kb":23518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2288},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'5968","reported_seq":6051,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:11.553954+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:11.553954+0000","last_peered":"2024-09-18T11:56:11.553954+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:11.553954+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5968,"log_dups_size":0,"ondisk_log_size":5968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24398244,"num_objects":5960,"num_object_clones":0,"num_object_copies":11920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5965,"num_write_kb":23831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2437},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":387596340,"num_objects":94645,"num_object_clones":0,"num_object_copies":189290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1755,"num_objects_unfound":0,"num_objects_dirty":94645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78700,"num_write_kb":314680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1745,"num_bytes_recovered":7066217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":781959168,"data_stored":781872193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94707,"ondisk_log_size":94707,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051612,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600364,"kb_used_data":302500,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93771476,"statfs":{"total":96636764160,"available":96021991424,"internally_reserved":0,"allocated":309760000,"data_stored":309546844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":1.5},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.28000000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.25900000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855375,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427004,"kb_used_data":120820,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93944836,"statfs":{"total":96636764160,"available":96199512064,"internally_reserved":0,"allocated":123719680,"data_stored":123426364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,5,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.443},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.375}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":4.6790000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.33400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637632,"kb_used_data":339828,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93734208,"statfs":{"total":96636764160,"available":95983828992,"internally_reserved":0,"allocated":347983872,"data_stored":347767094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,3,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.435},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.20100000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.47499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":348127232,"data_stored":348123753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":123437056,"data_stored":123357039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":310394880,"data_stored":310391401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:15.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 109 20 96214 96194 3.44634 2.03125 0.0244329 0.0226583 2024-09-18T11:56:16.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 110 20 96746 96726 3.4339 2.07812 0.052711 0.0227415 2024-09-18T11:56:17.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 111 20 97284 97264 3.4219 2.10156 0.0668998 0.0228236 2024-09-18T11:56:17.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:18.075 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:18.090 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:18.107 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-18T11:56:17.066640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":394285127,"num_objects":96280,"num_object_clones":0,"num_object_copies":192560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1755,"num_objects_unfound":0,"num_objects_dirty":96280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80335,"num_write_kb":321214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1746,"num_bytes_recovered":7066236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96343,"ondisk_log_size":96343,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1680872,"kb_used_data":779020,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281434648,"statfs":{"total":289910292480,"available":288189079552,"internally_reserved":0,"allocated":797716480,"data_stored":797000188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,7,5,5,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28008448,"num_objects":6838,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-304,"num_objects_unfound":0,"num_objects_dirty":6838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":27352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":304,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007508"},"pg_stats":[{"pgid":"2.f","version":"37'6135","reported_seq":6202,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.543575+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:16.543575+0000","last_peered":"2024-09-18T11:56:16.543575+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:16.543575+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25128960,"num_objects":6135,"num_object_clones":0,"num_object_copies":12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6135,"num_write_kb":24540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2536},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5897","reported_seq":6053,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:12.605239+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:12.605239+0000","last_peered":"2024-09-18T11:56:12.605239+0000","last_clean":"2024-09-18T11:56:12.605239+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:12.605239+0000","last_undegraded":"2024-09-18T11:56:12.605239+0000","last_fullsized":"2024-09-18T11:56:12.605239+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5897,"log_dups_size":0,"ondisk_log_size":5897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24129536,"num_objects":5891,"num_object_clones":0,"num_object_copies":11782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3089,"num_write_kb":12356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5941","reported_seq":6124,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:12.593225+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:12.593225+0000","last_peered":"2024-09-18T11:56:12.593225+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:12.593225+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24317952,"num_objects":5937,"num_object_clones":0,"num_object_copies":11874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2799,"num_write_kb":11196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2401}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5903","reported_seq":5930,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:12.579952+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:12.579952+0000","last_peered":"2024-09-18T11:56:12.579952+0000","last_clean":"2024-09-18T11:56:12.579952+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:12.579952+0000","last_undegraded":"2024-09-18T11:56:12.579952+0000","last_fullsized":"2024-09-18T11:56:12.579952+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5903,"log_dups_size":0,"ondisk_log_size":5903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24178688,"num_objects":5903,"num_object_clones":0,"num_object_copies":11806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5903,"num_write_kb":23612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6033","reported_seq":6222,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:16.542936+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:16.542936+0000","last_peered":"2024-09-18T11:56:16.542936+0000","last_clean":"2024-09-18T11:56:16.542936+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:16.542936+0000","last_undegraded":"2024-09-18T11:56:16.542936+0000","last_fullsized":"2024-09-18T11:56:16.542936+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24682496,"num_objects":6026,"num_object_clones":0,"num_object_copies":12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3198,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6019","reported_seq":6098,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.564958+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:16.564958+0000","last_peered":"2024-09-18T11:56:16.564958+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:16.564958+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24634578,"num_objects":6016,"num_object_clones":0,"num_object_copies":12032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":24059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2489},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6081","reported_seq":6166,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.578236+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:16.578236+0000","last_peered":"2024-09-18T11:56:16.578236+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:16.578236+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6081,"log_dups_size":0,"ondisk_log_size":6081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24884434,"num_objects":6077,"num_object_clones":0,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6079,"num_write_kb":24304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2518},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6100","reported_seq":6323,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:16.583782+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:16.583782+0000","last_peered":"2024-09-18T11:56:16.583782+0000","last_clean":"2024-09-18T11:56:16.583782+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:16.583782+0000","last_undegraded":"2024-09-18T11:56:16.583782+0000","last_fullsized":"2024-09-18T11:56:16.583782+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6100,"log_dups_size":0,"ondisk_log_size":6100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24977408,"num_objects":6098,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2417,"num_write_kb":9668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5888","reported_seq":9485,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:12.614050+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:12.614050+0000","last_peered":"2024-09-18T11:56:12.614050+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:12.614050+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5888,"log_dups_size":0,"ondisk_log_size":5888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24089810,"num_objects":5883,"num_object_clones":0,"num_object_copies":11766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1755,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5886,"num_write_kb":23528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1725,"num_bytes_recovered":7057408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1755},{"shards":"2","objects":4127}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6101","reported_seq":6177,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.488452+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:16.488452+0000","last_peered":"2024-09-18T11:56:16.488452+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:16.488452+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6101,"log_dups_size":0,"ondisk_log_size":6101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24958162,"num_objects":6095,"num_object_clones":0,"num_object_copies":12190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6099,"num_write_kb":24376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2563},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5989","reported_seq":6043,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:12.502608+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:12.502608+0000","last_peered":"2024-09-18T11:56:12.502608+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:12.502608+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5989,"log_dups_size":0,"ondisk_log_size":5989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24511698,"num_objects":5986,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5987,"num_write_kb":23939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2436}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5946","reported_seq":6153,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:12.552845+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:12.552845+0000","last_peered":"2024-09-18T11:56:12.552845+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:12.552845+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24330240,"num_objects":5940,"num_object_clones":0,"num_object_copies":11880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2426,"num_write_kb":9704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5997","reported_seq":6077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.511869+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:16.511869+0000","last_peered":"2024-09-18T11:56:16.511869+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:16.511869+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24556137,"num_objects":5996,"num_object_clones":0,"num_object_copies":11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5996,"num_write_kb":23981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2461},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6135","reported_seq":6217,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.536273+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:16.536273+0000","last_peered":"2024-09-18T11:56:16.536273+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:16.536273+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25098043,"num_objects":6130,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6132,"num_write_kb":24512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2608},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6056","reported_seq":6134,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.581245+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:16.581245+0000","last_peered":"2024-09-18T11:56:16.581245+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:16.581245+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6056,"log_dups_size":0,"ondisk_log_size":6056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24790226,"num_objects":6054,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6054,"num_write_kb":24210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2461},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6119","reported_seq":6201,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.552456+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:16.552456+0000","last_peered":"2024-09-18T11:56:16.552456+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:16.552456+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25016740,"num_objects":6111,"num_object_clones":0,"num_object_copies":12222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":24435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2588},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":394285108,"num_objects":96278,"num_object_clones":0,"num_object_copies":192556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1755,"num_objects_unfound":0,"num_objects_dirty":96278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80333,"num_write_kb":321212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1745,"num_bytes_recovered":7066217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":798416896,"data_stored":798336879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96340,"ondisk_log_size":96340,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051612,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600364,"kb_used_data":302500,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93771476,"statfs":{"total":96636764160,"available":96021991424,"internally_reserved":0,"allocated":309760000,"data_stored":309546844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":1.5},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.28000000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.25900000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426824,"kb_used_data":128956,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93945016,"statfs":{"total":96636764160,"available":96199696384,"internally_reserved":0,"allocated":132050944,"data_stored":131764586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.34799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":7.7629999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":6.492}]}]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653684,"kb_used_data":347564,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93718156,"statfs":{"total":96636764160,"available":95967391744,"internally_reserved":0,"allocated":355905536,"data_stored":355688758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.22},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.436},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.193}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356098048,"data_stored":356094569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131923968,"data_stored":131850909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":310394880,"data_stored":310391401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:18.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:18.122 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 112 20 97781 97761 3.40868 1.94141 0.0642391 0.0229117 2024-09-18T11:56:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:18.857 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:18.874 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-18T11:56:17.066640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":394285127,"num_objects":96280,"num_object_clones":0,"num_object_copies":192560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1755,"num_objects_unfound":0,"num_objects_dirty":96280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80335,"num_write_kb":321214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1746,"num_bytes_recovered":7066236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96343,"ondisk_log_size":96343,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1680872,"kb_used_data":779020,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281434648,"statfs":{"total":289910292480,"available":288189079552,"internally_reserved":0,"allocated":797716480,"data_stored":797000188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,7,5,5,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28008448,"num_objects":6838,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-304,"num_objects_unfound":0,"num_objects_dirty":6838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6838,"num_write_kb":27352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":304,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007508"},"pg_stats":[{"pgid":"2.f","version":"37'6135","reported_seq":6202,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.543575+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:16.543575+0000","last_peered":"2024-09-18T11:56:16.543575+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:16.543575+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25128960,"num_objects":6135,"num_object_clones":0,"num_object_copies":12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6135,"num_write_kb":24540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2536},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'5897","reported_seq":6053,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:12.605239+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:12.605239+0000","last_peered":"2024-09-18T11:56:12.605239+0000","last_clean":"2024-09-18T11:56:12.605239+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:12.605239+0000","last_undegraded":"2024-09-18T11:56:12.605239+0000","last_fullsized":"2024-09-18T11:56:12.605239+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":5897,"log_dups_size":0,"ondisk_log_size":5897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24129536,"num_objects":5891,"num_object_clones":0,"num_object_copies":11782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3089,"num_write_kb":12356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'5941","reported_seq":6124,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:12.593225+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:12.593225+0000","last_peered":"2024-09-18T11:56:12.593225+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:12.593225+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24317952,"num_objects":5937,"num_object_clones":0,"num_object_copies":11874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2799,"num_write_kb":11196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2401}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'5903","reported_seq":5930,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:12.579952+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:12.579952+0000","last_peered":"2024-09-18T11:56:12.579952+0000","last_clean":"2024-09-18T11:56:12.579952+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:12.579952+0000","last_undegraded":"2024-09-18T11:56:12.579952+0000","last_fullsized":"2024-09-18T11:56:12.579952+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5903,"log_dups_size":0,"ondisk_log_size":5903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24178688,"num_objects":5903,"num_object_clones":0,"num_object_copies":11806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5903,"num_write_kb":23612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6033","reported_seq":6222,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:16.542936+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:16.542936+0000","last_peered":"2024-09-18T11:56:16.542936+0000","last_clean":"2024-09-18T11:56:16.542936+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:16.542936+0000","last_undegraded":"2024-09-18T11:56:16.542936+0000","last_fullsized":"2024-09-18T11:56:16.542936+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24682496,"num_objects":6026,"num_object_clones":0,"num_object_copies":12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3198,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6019","reported_seq":6098,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.564958+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:16.564958+0000","last_peered":"2024-09-18T11:56:16.564958+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:16.564958+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24634578,"num_objects":6016,"num_object_clones":0,"num_object_copies":12032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":24059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2489},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6081","reported_seq":6166,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.578236+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:16.578236+0000","last_peered":"2024-09-18T11:56:16.578236+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:16.578236+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6081,"log_dups_size":0,"ondisk_log_size":6081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24884434,"num_objects":6077,"num_object_clones":0,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6079,"num_write_kb":24304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2518},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6100","reported_seq":6323,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:16.583782+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:16.583782+0000","last_peered":"2024-09-18T11:56:16.583782+0000","last_clean":"2024-09-18T11:56:16.583782+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:16.583782+0000","last_undegraded":"2024-09-18T11:56:16.583782+0000","last_fullsized":"2024-09-18T11:56:16.583782+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6100,"log_dups_size":0,"ondisk_log_size":6100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24977408,"num_objects":6098,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2417,"num_write_kb":9668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'5888","reported_seq":9485,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:12.614050+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:12.614050+0000","last_peered":"2024-09-18T11:56:12.614050+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:12.614050+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":5888,"log_dups_size":0,"ondisk_log_size":5888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24089810,"num_objects":5883,"num_object_clones":0,"num_object_copies":11766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1755,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5886,"num_write_kb":23528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1725,"num_bytes_recovered":7057408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1755},{"shards":"2","objects":4127}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6101","reported_seq":6177,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.488452+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:16.488452+0000","last_peered":"2024-09-18T11:56:16.488452+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:16.488452+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6101,"log_dups_size":0,"ondisk_log_size":6101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24958162,"num_objects":6095,"num_object_clones":0,"num_object_copies":12190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6099,"num_write_kb":24376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2563},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'5989","reported_seq":6043,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:12.502608+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:12.502608+0000","last_peered":"2024-09-18T11:56:12.502608+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:12.502608+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5989,"log_dups_size":0,"ondisk_log_size":5989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24511698,"num_objects":5986,"num_object_clones":0,"num_object_copies":11972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5987,"num_write_kb":23939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2436}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'5946","reported_seq":6153,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:12.552845+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:12.552845+0000","last_peered":"2024-09-18T11:56:12.552845+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:12.552845+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24330240,"num_objects":5940,"num_object_clones":0,"num_object_copies":11880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2426,"num_write_kb":9704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5997","reported_seq":6077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.511869+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:16.511869+0000","last_peered":"2024-09-18T11:56:16.511869+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:16.511869+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24556137,"num_objects":5996,"num_object_clones":0,"num_object_copies":11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5996,"num_write_kb":23981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2461},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6135","reported_seq":6217,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.536273+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:16.536273+0000","last_peered":"2024-09-18T11:56:16.536273+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:16.536273+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25098043,"num_objects":6130,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6132,"num_write_kb":24512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2608},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6056","reported_seq":6134,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.581245+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:16.581245+0000","last_peered":"2024-09-18T11:56:16.581245+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:16.581245+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6056,"log_dups_size":0,"ondisk_log_size":6056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24790226,"num_objects":6054,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6054,"num_write_kb":24210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2461},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6119","reported_seq":6201,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.552456+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:16.552456+0000","last_peered":"2024-09-18T11:56:16.552456+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:16.552456+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25016740,"num_objects":6111,"num_object_clones":0,"num_object_copies":12222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":24435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2588},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":394285108,"num_objects":96278,"num_object_clones":0,"num_object_copies":192556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1755,"num_objects_unfound":0,"num_objects_dirty":96278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80333,"num_write_kb":321212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1745,"num_bytes_recovered":7066217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":798416896,"data_stored":798336879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96340,"ondisk_log_size":96340,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051612,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600364,"kb_used_data":302500,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93771476,"statfs":{"total":96636764160,"available":96021991424,"internally_reserved":0,"allocated":309760000,"data_stored":309546844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":1.5},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.28000000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.25900000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426824,"kb_used_data":128956,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93945016,"statfs":{"total":96636764160,"available":96199696384,"internally_reserved":0,"allocated":132050944,"data_stored":131764586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.34799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":7.7629999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":6.492}]}]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653684,"kb_used_data":347564,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93718156,"statfs":{"total":96636764160,"available":95967391744,"internally_reserved":0,"allocated":355905536,"data_stored":355688758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.22},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.436},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.193}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356098048,"data_stored":356094569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131923968,"data_stored":131850909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":310394880,"data_stored":310391401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:18.875 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:56:18.875 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'6135", 'reported_seq': 6202, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:16.543575+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:56:16.543575+0000', 'last_peered': '2024-09-18T11:56:16.543575+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:56:16.543575+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6135, 'log_dups_size': 0, 'ondisk_log_size': 6135, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25128960, 'num_objects': 6135, 'num_object_clones': 0, 'num_object_copies': 12270, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6135, 'num_write_kb': 24540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2536}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:18.876 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:56:18.876 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'5941", 'reported_seq': 6124, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:12.593225+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:56:12.593225+0000', 'last_peered': '2024-09-18T11:56:12.593225+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:56:12.593225+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 5941, 'log_dups_size': 0, 'ondisk_log_size': 5941, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24317952, 'num_objects': 5937, 'num_object_clones': 0, 'num_object_copies': 11874, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2799, 'num_write_kb': 11196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 2401}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:18.876 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:56:18.876 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'6019", 'reported_seq': 6098, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:16.564958+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:56:16.564958+0000', 'last_peered': '2024-09-18T11:56:16.564958+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:56:16.564958+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6019, 'log_dups_size': 0, 'ondisk_log_size': 6019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24634578, 'num_objects': 6016, 'num_object_clones': 0, 'num_object_copies': 12032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6016, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6017, 'num_write_kb': 24059, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2489}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:18.876 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:56:18.876 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'6081", 'reported_seq': 6166, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:16.578236+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:56:16.578236+0000', 'last_peered': '2024-09-18T11:56:16.578236+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:56:16.578236+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6081, 'log_dups_size': 0, 'ondisk_log_size': 6081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24884434, 'num_objects': 6077, 'num_object_clones': 0, 'num_object_copies': 12154, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6079, 'num_write_kb': 24304, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2518}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:18.876 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:56:18.876 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'5888", 'reported_seq': 9485, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:12.614050+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:56:12.614050+0000', 'last_peered': '2024-09-18T11:56:12.614050+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:56:12.614050+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 5888, 'log_dups_size': 0, 'ondisk_log_size': 5888, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24089810, 'num_objects': 5883, 'num_object_clones': 0, 'num_object_copies': 11766, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 1755, 'num_objects_unfound': 0, 'num_objects_dirty': 5883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5886, 'num_write_kb': 23528, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1725, 'num_bytes_recovered': 7057408, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1755}, {'shards': '2', 'objects': 4127}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:18.876 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:56:18.876 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'6101", 'reported_seq': 6177, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:16.488452+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:56:16.488452+0000', 'last_peered': '2024-09-18T11:56:16.488452+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:56:16.488452+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6101, 'log_dups_size': 0, 'ondisk_log_size': 6101, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24958162, 'num_objects': 6095, 'num_object_clones': 0, 'num_object_copies': 12190, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6095, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6099, 'num_write_kb': 24376, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2563}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:18.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:56:18.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'5989", 'reported_seq': 6043, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:12.502608+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:56:12.502608+0000', 'last_peered': '2024-09-18T11:56:12.502608+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:56:12.502608+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5989, 'log_dups_size': 0, 'ondisk_log_size': 5989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24511698, 'num_objects': 5986, 'num_object_clones': 0, 'num_object_copies': 11972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5986, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5987, 'num_write_kb': 23939, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 2436}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:18.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:56:18.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'5946", 'reported_seq': 6153, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:12.552845+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:56:12.552845+0000', 'last_peered': '2024-09-18T11:56:12.552845+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:56:12.552845+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 5946, 'log_dups_size': 0, 'ondisk_log_size': 5946, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24330240, 'num_objects': 5940, 'num_object_clones': 0, 'num_object_copies': 11880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2426, 'num_write_kb': 9704, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 2347}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:18.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:56:18.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'5997", 'reported_seq': 6077, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:16.511869+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:56:16.511869+0000', 'last_peered': '2024-09-18T11:56:16.511869+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:56:16.511869+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 5997, 'log_dups_size': 0, 'ondisk_log_size': 5997, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24556137, 'num_objects': 5996, 'num_object_clones': 0, 'num_object_copies': 11992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5996, 'num_write_kb': 23981, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2461}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:18.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:56:18.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'6135", 'reported_seq': 6217, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:16.536273+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:56:16.536273+0000', 'last_peered': '2024-09-18T11:56:16.536273+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:56:16.536273+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6135, 'log_dups_size': 0, 'ondisk_log_size': 6135, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25098043, 'num_objects': 6130, 'num_object_clones': 0, 'num_object_copies': 12260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6132, 'num_write_kb': 24512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2608}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:18.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:56:18.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'6056", 'reported_seq': 6134, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:16.581245+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:56:16.581245+0000', 'last_peered': '2024-09-18T11:56:16.581245+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:56:16.581245+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6056, 'log_dups_size': 0, 'ondisk_log_size': 6056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24790226, 'num_objects': 6054, 'num_object_clones': 0, 'num_object_copies': 12108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6054, 'num_write_kb': 24210, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2461}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:18.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:56:18.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'6119", 'reported_seq': 6201, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:16.552456+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:56:16.552456+0000', 'last_peered': '2024-09-18T11:56:16.552456+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:56:16.552456+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6119, 'log_dups_size': 0, 'ondisk_log_size': 6119, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25016740, 'num_objects': 6111, 'num_object_clones': 0, 'num_object_copies': 12222, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6111, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6116, 'num_write_kb': 24435, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2588}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:18.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:19.123 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 113 20 98274 98254 3.39556 1.92578 0.0198098 0.0229988 2024-09-18T11:56:19.644 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:19.644 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/194470 objects degraded (20.101%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":179,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":97235,"data_bytes":398196807,"bytes_used":1730605056,"bytes_avail":288179687424,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":194470,"degraded_ratio":0.20100786753740937,"misplaced_objects":1600,"misplaced_total":194470,"misplaced_ratio":0.0082274901013009716,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":156570,"recovering_keys_per_sec":0,"num_objects_recovered":461,"num_bytes_recovered":1880064,"num_keys_recovered":0,"write_bytes_sec":2095108,"read_op_per_sec":0,"write_op_per_sec":511},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:19.680 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:19.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:20.126 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 114 20 98748 98728 3.3819 1.85156 0.0266543 0.0230917 2024-09-18T11:56:20.377 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:20.383 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:20.408 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-18T11:56:19.067400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398196807,"num_objects":97235,"num_object_clones":0,"num_object_copies":194470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1600,"num_objects_unfound":0,"num_objects_dirty":97235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81290,"num_write_kb":325034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1903,"num_bytes_recovered":7701116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97298,"ondisk_log_size":97298,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1690044,"kb_used_data":784032,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281425476,"statfs":{"total":289910292480,"available":288179687424,"internally_reserved":0,"allocated":802848768,"data_stored":802132476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,6,6,8,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25157632,"num_objects":6142,"num_object_clones":0,"num_object_copies":12284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-459,"num_objects_unfound":0,"num_objects_dirty":6142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6142,"num_write_kb":24568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":1880064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007793"},"pg_stats":[{"pgid":"2.f","version":"37'6135","reported_seq":6202,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.543575+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:16.543575+0000","last_peered":"2024-09-18T11:56:16.543575+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:16.543575+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25128960,"num_objects":6135,"num_object_clones":0,"num_object_copies":12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6135,"num_write_kb":24540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2536},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6068","reported_seq":6224,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:17.545970+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:17.545970+0000","last_peered":"2024-09-18T11:56:17.545970+0000","last_clean":"2024-09-18T11:56:17.545970+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:17.545970+0000","last_undegraded":"2024-09-18T11:56:17.545970+0000","last_fullsized":"2024-09-18T11:56:17.545970+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6068,"log_dups_size":0,"ondisk_log_size":6068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24829952,"num_objects":6062,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3260,"num_write_kb":13040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6090","reported_seq":6273,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:17.508985+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:17.508985+0000","last_peered":"2024-09-18T11:56:17.508985+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:17.508985+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6090,"log_dups_size":0,"ondisk_log_size":6090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24928256,"num_objects":6086,"num_object_clones":0,"num_object_copies":12172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2948,"num_write_kb":11792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2550}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6054","reported_seq":6081,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:17.546882+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:17.546882+0000","last_peered":"2024-09-18T11:56:17.546882+0000","last_clean":"2024-09-18T11:56:17.546882+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:17.546882+0000","last_undegraded":"2024-09-18T11:56:17.546882+0000","last_fullsized":"2024-09-18T11:56:17.546882+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6054,"log_dups_size":0,"ondisk_log_size":6054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24797184,"num_objects":6054,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6054,"num_write_kb":24216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6033","reported_seq":6222,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:16.542936+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:16.542936+0000","last_peered":"2024-09-18T11:56:16.542936+0000","last_clean":"2024-09-18T11:56:16.542936+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:16.542936+0000","last_undegraded":"2024-09-18T11:56:16.542936+0000","last_fullsized":"2024-09-18T11:56:16.542936+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24682496,"num_objects":6026,"num_object_clones":0,"num_object_copies":12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3198,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6019","reported_seq":6098,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.564958+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:16.564958+0000","last_peered":"2024-09-18T11:56:16.564958+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:16.564958+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24634578,"num_objects":6016,"num_object_clones":0,"num_object_copies":12032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":24059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2489},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6081","reported_seq":6166,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.578236+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:16.578236+0000","last_peered":"2024-09-18T11:56:16.578236+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:16.578236+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6081,"log_dups_size":0,"ondisk_log_size":6081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24884434,"num_objects":6077,"num_object_clones":0,"num_object_copies":12154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6079,"num_write_kb":24304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2518},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6100","reported_seq":6323,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:16.583782+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:16.583782+0000","last_peered":"2024-09-18T11:56:16.583782+0000","last_clean":"2024-09-18T11:56:16.583782+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:16.583782+0000","last_undegraded":"2024-09-18T11:56:16.583782+0000","last_fullsized":"2024-09-18T11:56:16.583782+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6100,"log_dups_size":0,"ondisk_log_size":6100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24977408,"num_objects":6098,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2417,"num_write_kb":9668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6056","reported_seq":9967,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:17.610265+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:17.610265+0000","last_peered":"2024-09-18T11:56:17.610265+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:17.610265+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6056,"log_dups_size":0,"ondisk_log_size":6056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24777938,"num_objects":6051,"num_object_clones":0,"num_object_copies":12102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1600,"num_objects_unfound":0,"num_objects_dirty":6051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6054,"num_write_kb":24200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1882,"num_bytes_recovered":7692288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1600},{"shards":"2","objects":4450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6101","reported_seq":6177,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.488452+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:16.488452+0000","last_peered":"2024-09-18T11:56:16.488452+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:16.488452+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6101,"log_dups_size":0,"ondisk_log_size":6101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24958162,"num_objects":6095,"num_object_clones":0,"num_object_copies":12190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6099,"num_write_kb":24376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2563},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6154","reported_seq":6208,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:17.554421+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:17.554421+0000","last_peered":"2024-09-18T11:56:17.554421+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:17.554421+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6154,"log_dups_size":0,"ondisk_log_size":6154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25187538,"num_objects":6151,"num_object_clones":0,"num_object_copies":12302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6152,"num_write_kb":24599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2601}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6097","reported_seq":6303,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:17.575773+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:17.575773+0000","last_peered":"2024-09-18T11:56:17.575773+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:17.575773+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6097,"log_dups_size":0,"ondisk_log_size":6097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24948736,"num_objects":6091,"num_object_clones":0,"num_object_copies":12182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":10308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2498}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'5997","reported_seq":6077,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.511869+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:16.511869+0000","last_peered":"2024-09-18T11:56:16.511869+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:16.511869+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24556137,"num_objects":5996,"num_object_clones":0,"num_object_copies":11992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5996,"num_write_kb":23981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2461},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6135","reported_seq":6217,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.536273+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:16.536273+0000","last_peered":"2024-09-18T11:56:16.536273+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:16.536273+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25098043,"num_objects":6130,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6132,"num_write_kb":24512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2608},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6056","reported_seq":6134,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.581245+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:16.581245+0000","last_peered":"2024-09-18T11:56:16.581245+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:16.581245+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6056,"log_dups_size":0,"ondisk_log_size":6056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24790226,"num_objects":6054,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6054,"num_write_kb":24210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2461},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6119","reported_seq":6201,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:16.552456+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:16.552456+0000","last_peered":"2024-09-18T11:56:16.552456+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:16.552456+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25016740,"num_objects":6111,"num_object_clones":0,"num_object_copies":12222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":24435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2588},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":398196788,"num_objects":97233,"num_object_clones":0,"num_object_copies":194466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1600,"num_objects_unfound":0,"num_objects_dirty":97233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81288,"num_write_kb":325032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1902,"num_bytes_recovered":7701097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":803733504,"data_stored":803653487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97295,"ondisk_log_size":97295,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051613,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609536,"kb_used_data":307512,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93762304,"statfs":{"total":96636764160,"available":96012599296,"internally_reserved":0,"allocated":314892288,"data_stored":314679132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.247},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":2.3359999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":2.2309999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426824,"kb_used_data":128956,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93945016,"statfs":{"total":96636764160,"available":96199696384,"internally_reserved":0,"allocated":132050944,"data_stored":131764586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.34799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":7.7629999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":6.492}]}]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653684,"kb_used_data":347564,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93718156,"statfs":{"total":96636764160,"available":95967391744,"internally_reserved":0,"allocated":355905536,"data_stored":355688758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.22},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.436},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.193}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356098048,"data_stored":356094569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131923968,"data_stored":131850909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":315711488,"data_stored":315708009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:21.126 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 115 20 99285 99265 3.37074 2.09766 0.0609499 0.0231704 2024-09-18T11:56:22.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 116 20 99818 99798 3.35963 2.08203 0.080581 0.0232462 2024-09-18T11:56:23.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 117 20 100348 100328 3.34861 2.07031 0.0138947 0.0233217 2024-09-18T11:56:23.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:24.013 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:24.055 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-18T11:56:23.068342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":408604743,"num_objects":99776,"num_object_clones":0,"num_object_copies":199552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1445,"num_objects_unfound":0,"num_objects_dirty":99776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83831,"num_write_kb":335198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2058,"num_bytes_recovered":8335996,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99839,"ondisk_log_size":99839,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1732964,"kb_used_data":806156,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281382556,"statfs":{"total":289910292480,"available":288135737344,"internally_reserved":0,"allocated":825503744,"data_stored":824787452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,6,10,13,10,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31502336,"num_objects":7691,"num_object_clones":0,"num_object_copies":15382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-468,"num_objects_unfound":0,"num_objects_dirty":7691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7691,"num_write_kb":30764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":1916928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007740"},"pg_stats":[{"pgid":"2.f","version":"37'6289","reported_seq":6358,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.578990+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:21.578990+0000","last_peered":"2024-09-18T11:56:21.578990+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:21.578990+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25759744,"num_objects":6289,"num_object_clones":0,"num_object_copies":12578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6289,"num_write_kb":25156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2690},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6255","reported_seq":6411,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:22.607927+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:22.607927+0000","last_peered":"2024-09-18T11:56:22.607927+0000","last_clean":"2024-09-18T11:56:22.607927+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:22.607927+0000","last_undegraded":"2024-09-18T11:56:22.607927+0000","last_fullsized":"2024-09-18T11:56:22.607927+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6255,"log_dups_size":0,"ondisk_log_size":6255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25595904,"num_objects":6249,"num_object_clones":0,"num_object_copies":12498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3447,"num_write_kb":13788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6239","reported_seq":6422,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:22.551657+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:22.551657+0000","last_peered":"2024-09-18T11:56:22.551657+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:22.551657+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6239,"log_dups_size":0,"ondisk_log_size":6239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25538560,"num_objects":6235,"num_object_clones":0,"num_object_copies":12470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3097,"num_write_kb":12388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6224","reported_seq":6251,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:22.565523+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:22.565523+0000","last_peered":"2024-09-18T11:56:22.565523+0000","last_clean":"2024-09-18T11:56:22.565523+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:22.565523+0000","last_undegraded":"2024-09-18T11:56:22.565523+0000","last_fullsized":"2024-09-18T11:56:22.565523+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25493504,"num_objects":6224,"num_object_clones":0,"num_object_copies":12448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6224,"num_write_kb":24896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6203","reported_seq":6392,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:21.601724+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:21.601724+0000","last_peered":"2024-09-18T11:56:21.601724+0000","last_clean":"2024-09-18T11:56:21.601724+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:21.601724+0000","last_undegraded":"2024-09-18T11:56:21.601724+0000","last_fullsized":"2024-09-18T11:56:21.601724+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6203,"log_dups_size":0,"ondisk_log_size":6203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25378816,"num_objects":6196,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3368,"num_write_kb":13472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6166","reported_seq":6245,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.541949+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:21.541949+0000","last_peered":"2024-09-18T11:56:21.541949+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:21.541949+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6166,"log_dups_size":0,"ondisk_log_size":6166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25236690,"num_objects":6163,"num_object_clones":0,"num_object_copies":12326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6164,"num_write_kb":24647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2636},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6254","reported_seq":6339,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.533041+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:21.533041+0000","last_peered":"2024-09-18T11:56:21.533041+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:21.533041+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6254,"log_dups_size":0,"ondisk_log_size":6254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25593042,"num_objects":6250,"num_object_clones":0,"num_object_copies":12500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6252,"num_write_kb":24996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2691},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6278","reported_seq":6501,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:21.546367+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:21.546367+0000","last_peered":"2024-09-18T11:56:21.546367+0000","last_clean":"2024-09-18T11:56:21.546367+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:21.546367+0000","last_undegraded":"2024-09-18T11:56:21.546367+0000","last_fullsized":"2024-09-18T11:56:21.546367+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6278,"log_dups_size":0,"ondisk_log_size":6278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25706496,"num_objects":6276,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2595,"num_write_kb":10380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6208","reported_seq":10430,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:22.613737+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:22.613737+0000","last_peered":"2024-09-18T11:56:22.613737+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:22.613737+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400530,"num_objects":6203,"num_object_clones":0,"num_object_copies":12406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1445,"num_objects_unfound":0,"num_objects_dirty":6203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":24808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2037,"num_bytes_recovered":8327168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1445},{"shards":"2","objects":4757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6267","reported_seq":6343,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.607881+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:21.607881+0000","last_peered":"2024-09-18T11:56:21.607881+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:21.607881+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25638098,"num_objects":6261,"num_object_clones":0,"num_object_copies":12522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6265,"num_write_kb":25040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2729},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6314","reported_seq":6368,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:22.579094+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:22.579094+0000","last_peered":"2024-09-18T11:56:22.579094+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:22.579094+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25842898,"num_objects":6311,"num_object_clones":0,"num_object_copies":12622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":25239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2761}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6258","reported_seq":6464,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:22.541843+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:22.541843+0000","last_peered":"2024-09-18T11:56:22.541843+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:22.541843+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6258,"log_dups_size":0,"ondisk_log_size":6258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25608192,"num_objects":6252,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2738,"num_write_kb":10952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2659}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6141","reported_seq":6221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.579568+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:21.579568+0000","last_peered":"2024-09-18T11:56:21.579568+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:21.579568+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6141,"log_dups_size":0,"ondisk_log_size":6141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25145961,"num_objects":6140,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6140,"num_write_kb":24557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2605},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6278","reported_seq":6360,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.634852+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:21.634852+0000","last_peered":"2024-09-18T11:56:21.634852+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:21.634852+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6278,"log_dups_size":0,"ondisk_log_size":6278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25683771,"num_objects":6273,"num_object_clones":0,"num_object_copies":12546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6275,"num_write_kb":25084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2751},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6193","reported_seq":6272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.493624+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:21.493624+0000","last_peered":"2024-09-18T11:56:21.493624+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:21.493624+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6193,"log_dups_size":0,"ondisk_log_size":6193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25351378,"num_objects":6191,"num_object_clones":0,"num_object_copies":12382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":24758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2598},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6269","reported_seq":6352,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.516505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:21.516505+0000","last_peered":"2024-09-18T11:56:21.516505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:21.516505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6269,"log_dups_size":0,"ondisk_log_size":6269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25631140,"num_objects":6261,"num_object_clones":0,"num_object_copies":12522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6266,"num_write_kb":25035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2738},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":408604724,"num_objects":99774,"num_object_clones":0,"num_object_copies":199548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1445,"num_objects_unfound":0,"num_objects_dirty":99774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83829,"num_write_kb":335196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":8335977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":825282560,"data_stored":825202543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99836,"ondisk_log_size":99836,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051614,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624264,"kb_used_data":313920,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93747576,"statfs":{"total":96636764160,"available":95997517824,"internally_reserved":0,"allocated":321454080,"data_stored":321240924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,9,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.31},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.372}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.35299999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855377,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443216,"kb_used_data":137032,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93928624,"statfs":{"total":96636764160,"available":96182910976,"internally_reserved":0,"allocated":140320768,"data_stored":140034410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.20000000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.39200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":665484,"kb_used_data":355204,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93706356,"statfs":{"total":96636764160,"available":95955308544,"internally_reserved":0,"allocated":363728896,"data_stored":363512118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.22},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.436},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.193}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":363945984,"data_stored":363942505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":140161024,"data_stored":140087965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321175552,"data_stored":321172073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:24.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:24.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 118 20 100835 100815 3.33636 1.90234 0.036193 0.023408 2024-09-18T11:56:24.786 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:24.795 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:24.829 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-18T11:56:23.068342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":408604743,"num_objects":99776,"num_object_clones":0,"num_object_copies":199552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1445,"num_objects_unfound":0,"num_objects_dirty":99776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83831,"num_write_kb":335198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2058,"num_bytes_recovered":8335996,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99839,"ondisk_log_size":99839,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1732964,"kb_used_data":806156,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281382556,"statfs":{"total":289910292480,"available":288135737344,"internally_reserved":0,"allocated":825503744,"data_stored":824787452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,6,10,13,10,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31502336,"num_objects":7691,"num_object_clones":0,"num_object_copies":15382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-468,"num_objects_unfound":0,"num_objects_dirty":7691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7691,"num_write_kb":30764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":1916928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007740"},"pg_stats":[{"pgid":"2.f","version":"37'6289","reported_seq":6358,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.578990+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:21.578990+0000","last_peered":"2024-09-18T11:56:21.578990+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:21.578990+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25759744,"num_objects":6289,"num_object_clones":0,"num_object_copies":12578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6289,"num_write_kb":25156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2690},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6255","reported_seq":6411,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:22.607927+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:22.607927+0000","last_peered":"2024-09-18T11:56:22.607927+0000","last_clean":"2024-09-18T11:56:22.607927+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:22.607927+0000","last_undegraded":"2024-09-18T11:56:22.607927+0000","last_fullsized":"2024-09-18T11:56:22.607927+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6255,"log_dups_size":0,"ondisk_log_size":6255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25595904,"num_objects":6249,"num_object_clones":0,"num_object_copies":12498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3447,"num_write_kb":13788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6239","reported_seq":6422,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:22.551657+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:22.551657+0000","last_peered":"2024-09-18T11:56:22.551657+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:22.551657+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6239,"log_dups_size":0,"ondisk_log_size":6239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25538560,"num_objects":6235,"num_object_clones":0,"num_object_copies":12470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3097,"num_write_kb":12388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6224","reported_seq":6251,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:22.565523+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:22.565523+0000","last_peered":"2024-09-18T11:56:22.565523+0000","last_clean":"2024-09-18T11:56:22.565523+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:22.565523+0000","last_undegraded":"2024-09-18T11:56:22.565523+0000","last_fullsized":"2024-09-18T11:56:22.565523+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25493504,"num_objects":6224,"num_object_clones":0,"num_object_copies":12448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6224,"num_write_kb":24896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6203","reported_seq":6392,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:21.601724+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:21.601724+0000","last_peered":"2024-09-18T11:56:21.601724+0000","last_clean":"2024-09-18T11:56:21.601724+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:21.601724+0000","last_undegraded":"2024-09-18T11:56:21.601724+0000","last_fullsized":"2024-09-18T11:56:21.601724+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6203,"log_dups_size":0,"ondisk_log_size":6203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25378816,"num_objects":6196,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3368,"num_write_kb":13472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6166","reported_seq":6245,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.541949+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:21.541949+0000","last_peered":"2024-09-18T11:56:21.541949+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:21.541949+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6166,"log_dups_size":0,"ondisk_log_size":6166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25236690,"num_objects":6163,"num_object_clones":0,"num_object_copies":12326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6164,"num_write_kb":24647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2636},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6254","reported_seq":6339,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.533041+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:21.533041+0000","last_peered":"2024-09-18T11:56:21.533041+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:21.533041+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6254,"log_dups_size":0,"ondisk_log_size":6254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25593042,"num_objects":6250,"num_object_clones":0,"num_object_copies":12500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6252,"num_write_kb":24996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2691},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6278","reported_seq":6501,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:21.546367+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:21.546367+0000","last_peered":"2024-09-18T11:56:21.546367+0000","last_clean":"2024-09-18T11:56:21.546367+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:21.546367+0000","last_undegraded":"2024-09-18T11:56:21.546367+0000","last_fullsized":"2024-09-18T11:56:21.546367+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6278,"log_dups_size":0,"ondisk_log_size":6278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25706496,"num_objects":6276,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2595,"num_write_kb":10380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6208","reported_seq":10430,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:22.613737+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:22.613737+0000","last_peered":"2024-09-18T11:56:22.613737+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:22.613737+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400530,"num_objects":6203,"num_object_clones":0,"num_object_copies":12406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1445,"num_objects_unfound":0,"num_objects_dirty":6203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":24808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2037,"num_bytes_recovered":8327168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1445},{"shards":"2","objects":4757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6267","reported_seq":6343,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.607881+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:21.607881+0000","last_peered":"2024-09-18T11:56:21.607881+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:21.607881+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25638098,"num_objects":6261,"num_object_clones":0,"num_object_copies":12522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6265,"num_write_kb":25040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2729},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6314","reported_seq":6368,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:22.579094+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:22.579094+0000","last_peered":"2024-09-18T11:56:22.579094+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:22.579094+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25842898,"num_objects":6311,"num_object_clones":0,"num_object_copies":12622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":25239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2761}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6258","reported_seq":6464,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:22.541843+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:22.541843+0000","last_peered":"2024-09-18T11:56:22.541843+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:22.541843+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6258,"log_dups_size":0,"ondisk_log_size":6258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25608192,"num_objects":6252,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2738,"num_write_kb":10952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2659}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6141","reported_seq":6221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.579568+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:21.579568+0000","last_peered":"2024-09-18T11:56:21.579568+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:21.579568+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6141,"log_dups_size":0,"ondisk_log_size":6141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25145961,"num_objects":6140,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6140,"num_write_kb":24557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2605},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6278","reported_seq":6360,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.634852+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:21.634852+0000","last_peered":"2024-09-18T11:56:21.634852+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:21.634852+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6278,"log_dups_size":0,"ondisk_log_size":6278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25683771,"num_objects":6273,"num_object_clones":0,"num_object_copies":12546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6275,"num_write_kb":25084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2751},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6193","reported_seq":6272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.493624+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:21.493624+0000","last_peered":"2024-09-18T11:56:21.493624+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:21.493624+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6193,"log_dups_size":0,"ondisk_log_size":6193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25351378,"num_objects":6191,"num_object_clones":0,"num_object_copies":12382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":24758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2598},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6269","reported_seq":6352,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.516505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:21.516505+0000","last_peered":"2024-09-18T11:56:21.516505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:21.516505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6269,"log_dups_size":0,"ondisk_log_size":6269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25631140,"num_objects":6261,"num_object_clones":0,"num_object_copies":12522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6266,"num_write_kb":25035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2738},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":408604724,"num_objects":99774,"num_object_clones":0,"num_object_copies":199548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1445,"num_objects_unfound":0,"num_objects_dirty":99774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83829,"num_write_kb":335196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":8335977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":825282560,"data_stored":825202543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99836,"ondisk_log_size":99836,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051614,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624264,"kb_used_data":313920,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93747576,"statfs":{"total":96636764160,"available":95997517824,"internally_reserved":0,"allocated":321454080,"data_stored":321240924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,9,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.31},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.372}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.35299999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855377,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443216,"kb_used_data":137032,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93928624,"statfs":{"total":96636764160,"available":96182910976,"internally_reserved":0,"allocated":140320768,"data_stored":140034410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.20000000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.39200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":665484,"kb_used_data":355204,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93706356,"statfs":{"total":96636764160,"available":95955308544,"internally_reserved":0,"allocated":363728896,"data_stored":363512118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.22},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.436},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.193}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":363945984,"data_stored":363942505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":140161024,"data_stored":140087965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321175552,"data_stored":321172073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:24.830 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:56:24.830 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'6289", 'reported_seq': 6358, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:21.578990+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:56:21.578990+0000', 'last_peered': '2024-09-18T11:56:21.578990+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:56:21.578990+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6289, 'log_dups_size': 0, 'ondisk_log_size': 6289, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25759744, 'num_objects': 6289, 'num_object_clones': 0, 'num_object_copies': 12578, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6289, 'num_write_kb': 25156, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2690}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:24.830 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:56:24.830 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'6239", 'reported_seq': 6422, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:22.551657+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:56:22.551657+0000', 'last_peered': '2024-09-18T11:56:22.551657+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:56:22.551657+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 6239, 'log_dups_size': 0, 'ondisk_log_size': 6239, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25538560, 'num_objects': 6235, 'num_object_clones': 0, 'num_object_copies': 12470, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3097, 'num_write_kb': 12388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 2699}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:24.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:56:24.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'6166", 'reported_seq': 6245, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:21.541949+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:56:21.541949+0000', 'last_peered': '2024-09-18T11:56:21.541949+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:56:21.541949+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6166, 'log_dups_size': 0, 'ondisk_log_size': 6166, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25236690, 'num_objects': 6163, 'num_object_clones': 0, 'num_object_copies': 12326, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6164, 'num_write_kb': 24647, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2636}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:24.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:56:24.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'6254", 'reported_seq': 6339, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:21.533041+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:56:21.533041+0000', 'last_peered': '2024-09-18T11:56:21.533041+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:56:21.533041+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6254, 'log_dups_size': 0, 'ondisk_log_size': 6254, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25593042, 'num_objects': 6250, 'num_object_clones': 0, 'num_object_copies': 12500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6252, 'num_write_kb': 24996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2691}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:24.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:56:24.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'6208", 'reported_seq': 10430, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:22.613737+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:56:22.613737+0000', 'last_peered': '2024-09-18T11:56:22.613737+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:56:22.613737+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 6208, 'log_dups_size': 0, 'ondisk_log_size': 6208, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25400530, 'num_objects': 6203, 'num_object_clones': 0, 'num_object_copies': 12406, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 1445, 'num_objects_unfound': 0, 'num_objects_dirty': 6203, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6206, 'num_write_kb': 24808, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2037, 'num_bytes_recovered': 8327168, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1445}, {'shards': '2', 'objects': 4757}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:24.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:56:24.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'6267", 'reported_seq': 6343, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:21.607881+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:56:21.607881+0000', 'last_peered': '2024-09-18T11:56:21.607881+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:56:21.607881+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6267, 'log_dups_size': 0, 'ondisk_log_size': 6267, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25638098, 'num_objects': 6261, 'num_object_clones': 0, 'num_object_copies': 12522, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6265, 'num_write_kb': 25040, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2729}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:24.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:56:24.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'6314", 'reported_seq': 6368, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:22.579094+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:56:22.579094+0000', 'last_peered': '2024-09-18T11:56:22.579094+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:56:22.579094+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6314, 'log_dups_size': 0, 'ondisk_log_size': 6314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25842898, 'num_objects': 6311, 'num_object_clones': 0, 'num_object_copies': 12622, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6312, 'num_write_kb': 25239, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 2761}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:24.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:56:24.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'6258", 'reported_seq': 6464, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:22.541843+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:56:22.541843+0000', 'last_peered': '2024-09-18T11:56:22.541843+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:56:22.541843+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 6258, 'log_dups_size': 0, 'ondisk_log_size': 6258, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25608192, 'num_objects': 6252, 'num_object_clones': 0, 'num_object_copies': 12504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2738, 'num_write_kb': 10952, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 2659}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:24.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:56:24.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'6141", 'reported_seq': 6221, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:21.579568+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:56:21.579568+0000', 'last_peered': '2024-09-18T11:56:21.579568+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:56:21.579568+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6141, 'log_dups_size': 0, 'ondisk_log_size': 6141, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25145961, 'num_objects': 6140, 'num_object_clones': 0, 'num_object_copies': 12280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6140, 'num_write_kb': 24557, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2605}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:24.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:56:24.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'6278", 'reported_seq': 6360, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:21.634852+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:56:21.634852+0000', 'last_peered': '2024-09-18T11:56:21.634852+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:56:21.634852+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6278, 'log_dups_size': 0, 'ondisk_log_size': 6278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25683771, 'num_objects': 6273, 'num_object_clones': 0, 'num_object_copies': 12546, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6275, 'num_write_kb': 25084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2751}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:24.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:56:24.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'6193", 'reported_seq': 6272, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:21.493624+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:56:21.493624+0000', 'last_peered': '2024-09-18T11:56:21.493624+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:56:21.493624+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6193, 'log_dups_size': 0, 'ondisk_log_size': 6193, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25351378, 'num_objects': 6191, 'num_object_clones': 0, 'num_object_copies': 12382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6191, 'num_write_kb': 24758, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2598}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:24.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:56:24.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'6269", 'reported_seq': 6352, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:21.516505+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:56:21.516505+0000', 'last_peered': '2024-09-18T11:56:21.516505+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:56:21.516505+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6269, 'log_dups_size': 0, 'ondisk_log_size': 6269, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25631140, 'num_objects': 6261, 'num_object_clones': 0, 'num_object_copies': 12522, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6266, 'num_write_kb': 25035, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2738}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:24.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:25.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 119 20 101331 101311 3.3246 1.9375 0.0424542 0.0234904 2024-09-18T11:56:25.638 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:25.638 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/199552 objects degraded (19.589%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":185,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":99776,"data_bytes":408604743,"bytes_used":1774555136,"bytes_avail":288135737344,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":199552,"degraded_ratio":0.19588879089159719,"misplaced_objects":1445,"misplaced_total":199552,"misplaced_ratio":0.0072412203335471444,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":105773,"recovering_keys_per_sec":0,"num_objects_recovered":312,"num_bytes_recovered":1269760,"num_keys_recovered":0,"write_bytes_sec":1750038,"read_op_per_sec":0,"write_op_per_sec":427},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:25.674 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:25.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:26.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:2024-09-18T11:56:26.127891+0000 min lat: 0.0021114 max lat: 2.57663 avg lat: 0.0235733 2024-09-18T11:56:26.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:56:26.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 120 20 101806 101786 3.31236 1.85547 0.0259493 0.0235733 2024-09-18T11:56:26.260 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:26.269 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:26.287 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-18T11:56:25.068880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":408604743,"num_objects":99776,"num_object_clones":0,"num_object_copies":199552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1445,"num_objects_unfound":0,"num_objects_dirty":99776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83831,"num_write_kb":335198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2058,"num_bytes_recovered":8335996,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99839,"ondisk_log_size":99839,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1732964,"kb_used_data":806156,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281382556,"statfs":{"total":289910292480,"available":288135737344,"internally_reserved":0,"allocated":825503744,"data_stored":824787452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,6,10,13,10,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21008384,"num_objects":5129,"num_object_clones":0,"num_object_copies":10258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":5129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5129,"num_write_kb":20516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004527"},"pg_stats":[{"pgid":"2.f","version":"37'6289","reported_seq":6358,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.578990+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:21.578990+0000","last_peered":"2024-09-18T11:56:21.578990+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:21.578990+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25759744,"num_objects":6289,"num_object_clones":0,"num_object_copies":12578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6289,"num_write_kb":25156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2690},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6255","reported_seq":6411,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:22.607927+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:22.607927+0000","last_peered":"2024-09-18T11:56:22.607927+0000","last_clean":"2024-09-18T11:56:22.607927+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:22.607927+0000","last_undegraded":"2024-09-18T11:56:22.607927+0000","last_fullsized":"2024-09-18T11:56:22.607927+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6255,"log_dups_size":0,"ondisk_log_size":6255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25595904,"num_objects":6249,"num_object_clones":0,"num_object_copies":12498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3447,"num_write_kb":13788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6239","reported_seq":6422,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:22.551657+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:22.551657+0000","last_peered":"2024-09-18T11:56:22.551657+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:22.551657+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6239,"log_dups_size":0,"ondisk_log_size":6239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25538560,"num_objects":6235,"num_object_clones":0,"num_object_copies":12470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3097,"num_write_kb":12388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6224","reported_seq":6251,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:22.565523+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:22.565523+0000","last_peered":"2024-09-18T11:56:22.565523+0000","last_clean":"2024-09-18T11:56:22.565523+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:22.565523+0000","last_undegraded":"2024-09-18T11:56:22.565523+0000","last_fullsized":"2024-09-18T11:56:22.565523+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25493504,"num_objects":6224,"num_object_clones":0,"num_object_copies":12448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6224,"num_write_kb":24896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6203","reported_seq":6392,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:21.601724+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:21.601724+0000","last_peered":"2024-09-18T11:56:21.601724+0000","last_clean":"2024-09-18T11:56:21.601724+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:21.601724+0000","last_undegraded":"2024-09-18T11:56:21.601724+0000","last_fullsized":"2024-09-18T11:56:21.601724+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6203,"log_dups_size":0,"ondisk_log_size":6203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25378816,"num_objects":6196,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3368,"num_write_kb":13472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6166","reported_seq":6245,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.541949+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:21.541949+0000","last_peered":"2024-09-18T11:56:21.541949+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:21.541949+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6166,"log_dups_size":0,"ondisk_log_size":6166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25236690,"num_objects":6163,"num_object_clones":0,"num_object_copies":12326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6164,"num_write_kb":24647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2636},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6254","reported_seq":6339,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.533041+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:21.533041+0000","last_peered":"2024-09-18T11:56:21.533041+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:21.533041+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6254,"log_dups_size":0,"ondisk_log_size":6254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25593042,"num_objects":6250,"num_object_clones":0,"num_object_copies":12500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6252,"num_write_kb":24996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2691},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6278","reported_seq":6501,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:21.546367+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:21.546367+0000","last_peered":"2024-09-18T11:56:21.546367+0000","last_clean":"2024-09-18T11:56:21.546367+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:21.546367+0000","last_undegraded":"2024-09-18T11:56:21.546367+0000","last_fullsized":"2024-09-18T11:56:21.546367+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6278,"log_dups_size":0,"ondisk_log_size":6278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25706496,"num_objects":6276,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2595,"num_write_kb":10380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6208","reported_seq":10430,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:22.613737+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:22.613737+0000","last_peered":"2024-09-18T11:56:22.613737+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:22.613737+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400530,"num_objects":6203,"num_object_clones":0,"num_object_copies":12406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1445,"num_objects_unfound":0,"num_objects_dirty":6203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":24808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2037,"num_bytes_recovered":8327168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1445},{"shards":"2","objects":4757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6267","reported_seq":6343,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.607881+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:21.607881+0000","last_peered":"2024-09-18T11:56:21.607881+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:21.607881+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25638098,"num_objects":6261,"num_object_clones":0,"num_object_copies":12522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6265,"num_write_kb":25040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2729},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6314","reported_seq":6368,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:22.579094+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:22.579094+0000","last_peered":"2024-09-18T11:56:22.579094+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:22.579094+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25842898,"num_objects":6311,"num_object_clones":0,"num_object_copies":12622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":25239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2761}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6258","reported_seq":6464,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:22.541843+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:22.541843+0000","last_peered":"2024-09-18T11:56:22.541843+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:22.541843+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6258,"log_dups_size":0,"ondisk_log_size":6258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25608192,"num_objects":6252,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2738,"num_write_kb":10952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2659}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6141","reported_seq":6221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.579568+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:21.579568+0000","last_peered":"2024-09-18T11:56:21.579568+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:21.579568+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6141,"log_dups_size":0,"ondisk_log_size":6141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25145961,"num_objects":6140,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6140,"num_write_kb":24557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2605},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6278","reported_seq":6360,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.634852+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:21.634852+0000","last_peered":"2024-09-18T11:56:21.634852+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:21.634852+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6278,"log_dups_size":0,"ondisk_log_size":6278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25683771,"num_objects":6273,"num_object_clones":0,"num_object_copies":12546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6275,"num_write_kb":25084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2751},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6193","reported_seq":6272,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.493624+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:21.493624+0000","last_peered":"2024-09-18T11:56:21.493624+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:21.493624+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6193,"log_dups_size":0,"ondisk_log_size":6193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25351378,"num_objects":6191,"num_object_clones":0,"num_object_copies":12382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":24758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2598},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6269","reported_seq":6352,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:21.516505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:21.516505+0000","last_peered":"2024-09-18T11:56:21.516505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:21.516505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6269,"log_dups_size":0,"ondisk_log_size":6269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25631140,"num_objects":6261,"num_object_clones":0,"num_object_copies":12522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6266,"num_write_kb":25035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2738},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":408604724,"num_objects":99774,"num_object_clones":0,"num_object_copies":199548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1445,"num_objects_unfound":0,"num_objects_dirty":99774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83829,"num_write_kb":335196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":8335977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":825282560,"data_stored":825202543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99836,"ondisk_log_size":99836,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051614,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624264,"kb_used_data":313920,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93747576,"statfs":{"total":96636764160,"available":95997517824,"internally_reserved":0,"allocated":321454080,"data_stored":321240924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,9,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.31},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.372}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.35299999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855377,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443216,"kb_used_data":137032,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93928624,"statfs":{"total":96636764160,"available":96182910976,"internally_reserved":0,"allocated":140320768,"data_stored":140034410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.20000000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.39200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":665484,"kb_used_data":355204,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93706356,"statfs":{"total":96636764160,"available":95955308544,"internally_reserved":0,"allocated":363728896,"data_stored":363512118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.22},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":0.436},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.193}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":363945984,"data_stored":363942505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":140161024,"data_stored":140087965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321175552,"data_stored":321172073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:27.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 121 20 102328 102308 3.30184 2.03906 0.0204028 0.0236527 2024-09-18T11:56:28.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 122 20 102856 102836 3.29168 2.0625 0.0517122 0.0237273 2024-09-18T11:56:29.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 123 20 103378 103358 3.2815 2.03906 0.0424065 0.0237986 2024-09-18T11:56:29.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:30.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 124 20 103875 103855 3.27069 1.94141 0.0354224 0.0238788 2024-09-18T11:56:30.153 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:30.171 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:30.185 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-18T11:56:29.069819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":418918471,"num_objects":102294,"num_object_clones":0,"num_object_copies":204588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1271,"num_objects_unfound":0,"num_objects_dirty":102294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86349,"num_write_kb":345270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2232,"num_bytes_recovered":9048700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102357,"ondisk_log_size":102357,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1737300,"kb_used_data":827132,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281378220,"statfs":{"total":289910292480,"available":288131297280,"internally_reserved":0,"allocated":846983168,"data_stored":846266876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,10,17,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24633344,"num_objects":6014,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-484,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6014,"num_write_kb":24056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":1982464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003178"},"pg_stats":[{"pgid":"2.f","version":"37'6441","reported_seq":6510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.627721+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:26.627721+0000","last_peered":"2024-09-18T11:56:26.627721+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:26.627721+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6441,"log_dups_size":0,"ondisk_log_size":6441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26382336,"num_objects":6441,"num_object_clones":0,"num_object_copies":12882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6441,"num_write_kb":25764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2842},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6420","reported_seq":6576,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:27.604965+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:27.604965+0000","last_peered":"2024-09-18T11:56:27.604965+0000","last_clean":"2024-09-18T11:56:27.604965+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:27.604965+0000","last_undegraded":"2024-09-18T11:56:27.604965+0000","last_fullsized":"2024-09-18T11:56:27.604965+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6420,"log_dups_size":0,"ondisk_log_size":6420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26271744,"num_objects":6414,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3612,"num_write_kb":14448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6391","reported_seq":6574,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:27.580031+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:27.580031+0000","last_peered":"2024-09-18T11:56:27.580031+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:27.580031+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26161152,"num_objects":6387,"num_object_clones":0,"num_object_copies":12774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3249,"num_write_kb":12996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6377","reported_seq":6404,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:27.607886+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:27.607886+0000","last_peered":"2024-09-18T11:56:27.607886+0000","last_clean":"2024-09-18T11:56:27.607886+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:27.607886+0000","last_undegraded":"2024-09-18T11:56:27.607886+0000","last_fullsized":"2024-09-18T11:56:27.607886+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6377,"log_dups_size":0,"ondisk_log_size":6377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26120192,"num_objects":6377,"num_object_clones":0,"num_object_copies":12754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6377,"num_write_kb":25508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6372","reported_seq":6561,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:26.592089+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:26.592089+0000","last_peered":"2024-09-18T11:56:26.592089+0000","last_clean":"2024-09-18T11:56:26.592089+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:26.592089+0000","last_undegraded":"2024-09-18T11:56:26.592089+0000","last_fullsized":"2024-09-18T11:56:26.592089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6372,"log_dups_size":0,"ondisk_log_size":6372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26071040,"num_objects":6365,"num_object_clones":0,"num_object_copies":12730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6317","reported_seq":6396,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.575057+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:26.575057+0000","last_peered":"2024-09-18T11:56:26.575057+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:26.575057+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6317,"log_dups_size":0,"ondisk_log_size":6317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25855186,"num_objects":6314,"num_object_clones":0,"num_object_copies":12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6315,"num_write_kb":25251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2787},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6410","reported_seq":6493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.563712+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:26.563712+0000","last_peered":"2024-09-18T11:56:26.563712+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:26.563712+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26232018,"num_objects":6406,"num_object_clones":0,"num_object_copies":12812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6408,"num_write_kb":25620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2847},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6458","reported_seq":6681,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:26.644227+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:26.644227+0000","last_peered":"2024-09-18T11:56:26.644227+0000","last_clean":"2024-09-18T11:56:26.644227+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:26.644227+0000","last_undegraded":"2024-09-18T11:56:26.644227+0000","last_fullsized":"2024-09-18T11:56:26.644227+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6458,"log_dups_size":0,"ondisk_log_size":6458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26443776,"num_objects":6456,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2775,"num_write_kb":11100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6377","reported_seq":10941,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:27.617250+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:27.617250+0000","last_peered":"2024-09-18T11:56:27.617250+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:27.617250+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6377,"log_dups_size":0,"ondisk_log_size":6377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26092754,"num_objects":6372,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1271,"num_objects_unfound":0,"num_objects_dirty":6372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6375,"num_write_kb":25484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2211,"num_bytes_recovered":9039872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1271},{"shards":"2","objects":5100}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6415","reported_seq":6491,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.572927+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:26.572927+0000","last_peered":"2024-09-18T11:56:26.572927+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:26.572927+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26244306,"num_objects":6409,"num_object_clones":0,"num_object_copies":12818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6413,"num_write_kb":25632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2877},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6449","reported_seq":6503,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:27.628227+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:27.628227+0000","last_peered":"2024-09-18T11:56:27.628227+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:27.628227+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26395858,"num_objects":6446,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":25779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6422","reported_seq":6629,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:27.577938+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:27.577938+0000","last_peered":"2024-09-18T11:56:27.577938+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:27.577938+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26279936,"num_objects":6416,"num_object_clones":0,"num_object_copies":12832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2823}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6292","reported_seq":6372,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.540544+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:26.540544+0000","last_peered":"2024-09-18T11:56:26.540544+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:26.540544+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25764457,"num_objects":6291,"num_object_clones":0,"num_object_copies":12582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":25161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2756},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6436","reported_seq":6518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.662542+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:26.662542+0000","last_peered":"2024-09-18T11:56:26.662542+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:26.662542+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26330939,"num_objects":6431,"num_object_clones":0,"num_object_copies":12862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6433,"num_write_kb":25716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2909},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6351","reported_seq":6430,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.591795+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:26.591795+0000","last_peered":"2024-09-18T11:56:26.591795+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:26.591795+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6351,"log_dups_size":0,"ondisk_log_size":6351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25998546,"num_objects":6349,"num_object_clones":0,"num_object_copies":12698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":25390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2756},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6426","reported_seq":6509,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.655640+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:26.655640+0000","last_peered":"2024-09-18T11:56:26.655640+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:26.655640+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26274212,"num_objects":6418,"num_object_clones":0,"num_object_copies":12836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6423,"num_write_kb":25663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":418918452,"num_objects":102292,"num_object_clones":0,"num_object_copies":204584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1271,"num_objects_unfound":0,"num_objects_dirty":102292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86347,"num_write_kb":345268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2231,"num_bytes_recovered":9048681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":846569472,"data_stored":846489455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102354,"ondisk_log_size":102354,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051615,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616812,"kb_used_data":318948,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93755028,"statfs":{"total":96636764160,"available":96005148672,"internally_reserved":0,"allocated":326602752,"data_stored":326389596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.439}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.45900000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855378,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455596,"kb_used_data":145252,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93916244,"statfs":{"total":96636764160,"available":96170233856,"internally_reserved":0,"allocated":148738048,"data_stored":148451690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.24099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":4.1340000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.39500000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664892,"kb_used_data":362932,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93706948,"statfs":{"total":96636764160,"available":95955914752,"internally_reserved":0,"allocated":371642368,"data_stored":371425590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,7,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":2.7770000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":371703808,"data_stored":371700329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148480000,"data_stored":148406941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":326385664,"data_stored":326382185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:30.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:30.830 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:30.851 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:30.881 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-18T11:56:29.069819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":418918471,"num_objects":102294,"num_object_clones":0,"num_object_copies":204588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1271,"num_objects_unfound":0,"num_objects_dirty":102294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86349,"num_write_kb":345270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2232,"num_bytes_recovered":9048700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102357,"ondisk_log_size":102357,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1737300,"kb_used_data":827132,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281378220,"statfs":{"total":289910292480,"available":288131297280,"internally_reserved":0,"allocated":846983168,"data_stored":846266876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,10,17,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24633344,"num_objects":6014,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-484,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6014,"num_write_kb":24056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":1982464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003178"},"pg_stats":[{"pgid":"2.f","version":"37'6441","reported_seq":6510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.627721+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:26.627721+0000","last_peered":"2024-09-18T11:56:26.627721+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:26.627721+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6441,"log_dups_size":0,"ondisk_log_size":6441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26382336,"num_objects":6441,"num_object_clones":0,"num_object_copies":12882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6441,"num_write_kb":25764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2842},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6420","reported_seq":6576,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:27.604965+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:27.604965+0000","last_peered":"2024-09-18T11:56:27.604965+0000","last_clean":"2024-09-18T11:56:27.604965+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:27.604965+0000","last_undegraded":"2024-09-18T11:56:27.604965+0000","last_fullsized":"2024-09-18T11:56:27.604965+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6420,"log_dups_size":0,"ondisk_log_size":6420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26271744,"num_objects":6414,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3612,"num_write_kb":14448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6391","reported_seq":6574,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:27.580031+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:27.580031+0000","last_peered":"2024-09-18T11:56:27.580031+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:27.580031+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26161152,"num_objects":6387,"num_object_clones":0,"num_object_copies":12774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3249,"num_write_kb":12996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6377","reported_seq":6404,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:27.607886+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:27.607886+0000","last_peered":"2024-09-18T11:56:27.607886+0000","last_clean":"2024-09-18T11:56:27.607886+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:27.607886+0000","last_undegraded":"2024-09-18T11:56:27.607886+0000","last_fullsized":"2024-09-18T11:56:27.607886+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6377,"log_dups_size":0,"ondisk_log_size":6377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26120192,"num_objects":6377,"num_object_clones":0,"num_object_copies":12754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6377,"num_write_kb":25508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6372","reported_seq":6561,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:26.592089+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:26.592089+0000","last_peered":"2024-09-18T11:56:26.592089+0000","last_clean":"2024-09-18T11:56:26.592089+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:26.592089+0000","last_undegraded":"2024-09-18T11:56:26.592089+0000","last_fullsized":"2024-09-18T11:56:26.592089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6372,"log_dups_size":0,"ondisk_log_size":6372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26071040,"num_objects":6365,"num_object_clones":0,"num_object_copies":12730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6317","reported_seq":6396,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.575057+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:26.575057+0000","last_peered":"2024-09-18T11:56:26.575057+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:26.575057+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6317,"log_dups_size":0,"ondisk_log_size":6317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25855186,"num_objects":6314,"num_object_clones":0,"num_object_copies":12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6315,"num_write_kb":25251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2787},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6410","reported_seq":6493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.563712+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:26.563712+0000","last_peered":"2024-09-18T11:56:26.563712+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:26.563712+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26232018,"num_objects":6406,"num_object_clones":0,"num_object_copies":12812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6408,"num_write_kb":25620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2847},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6458","reported_seq":6681,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:26.644227+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:26.644227+0000","last_peered":"2024-09-18T11:56:26.644227+0000","last_clean":"2024-09-18T11:56:26.644227+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:26.644227+0000","last_undegraded":"2024-09-18T11:56:26.644227+0000","last_fullsized":"2024-09-18T11:56:26.644227+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6458,"log_dups_size":0,"ondisk_log_size":6458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26443776,"num_objects":6456,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2775,"num_write_kb":11100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6377","reported_seq":10941,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:27.617250+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:27.617250+0000","last_peered":"2024-09-18T11:56:27.617250+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:27.617250+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6377,"log_dups_size":0,"ondisk_log_size":6377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26092754,"num_objects":6372,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1271,"num_objects_unfound":0,"num_objects_dirty":6372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6375,"num_write_kb":25484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2211,"num_bytes_recovered":9039872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1271},{"shards":"2","objects":5100}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6415","reported_seq":6491,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.572927+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:26.572927+0000","last_peered":"2024-09-18T11:56:26.572927+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:26.572927+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26244306,"num_objects":6409,"num_object_clones":0,"num_object_copies":12818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6413,"num_write_kb":25632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2877},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6449","reported_seq":6503,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:27.628227+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:27.628227+0000","last_peered":"2024-09-18T11:56:27.628227+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:27.628227+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26395858,"num_objects":6446,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":25779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6422","reported_seq":6629,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:27.577938+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:27.577938+0000","last_peered":"2024-09-18T11:56:27.577938+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:27.577938+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26279936,"num_objects":6416,"num_object_clones":0,"num_object_copies":12832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2823}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6292","reported_seq":6372,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.540544+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:26.540544+0000","last_peered":"2024-09-18T11:56:26.540544+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:26.540544+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25764457,"num_objects":6291,"num_object_clones":0,"num_object_copies":12582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":25161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2756},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6436","reported_seq":6518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.662542+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:26.662542+0000","last_peered":"2024-09-18T11:56:26.662542+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:26.662542+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26330939,"num_objects":6431,"num_object_clones":0,"num_object_copies":12862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6433,"num_write_kb":25716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2909},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6351","reported_seq":6430,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.591795+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:26.591795+0000","last_peered":"2024-09-18T11:56:26.591795+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:26.591795+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6351,"log_dups_size":0,"ondisk_log_size":6351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25998546,"num_objects":6349,"num_object_clones":0,"num_object_copies":12698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":25390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2756},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6426","reported_seq":6509,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.655640+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:26.655640+0000","last_peered":"2024-09-18T11:56:26.655640+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:26.655640+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26274212,"num_objects":6418,"num_object_clones":0,"num_object_copies":12836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6423,"num_write_kb":25663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":418918452,"num_objects":102292,"num_object_clones":0,"num_object_copies":204584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1271,"num_objects_unfound":0,"num_objects_dirty":102292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86347,"num_write_kb":345268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2231,"num_bytes_recovered":9048681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":846569472,"data_stored":846489455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102354,"ondisk_log_size":102354,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051615,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616812,"kb_used_data":318948,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93755028,"statfs":{"total":96636764160,"available":96005148672,"internally_reserved":0,"allocated":326602752,"data_stored":326389596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.439}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.45900000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855378,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455596,"kb_used_data":145252,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93916244,"statfs":{"total":96636764160,"available":96170233856,"internally_reserved":0,"allocated":148738048,"data_stored":148451690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.24099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":4.1340000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.39500000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664892,"kb_used_data":362932,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93706948,"statfs":{"total":96636764160,"available":95955914752,"internally_reserved":0,"allocated":371642368,"data_stored":371425590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,7,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":2.7770000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":371703808,"data_stored":371700329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148480000,"data_stored":148406941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":326385664,"data_stored":326382185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:30.883 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:56:30.883 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'6441", 'reported_seq': 6510, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:26.627721+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:56:26.627721+0000', 'last_peered': '2024-09-18T11:56:26.627721+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:56:26.627721+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6441, 'log_dups_size': 0, 'ondisk_log_size': 6441, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26382336, 'num_objects': 6441, 'num_object_clones': 0, 'num_object_copies': 12882, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6441, 'num_write_kb': 25764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2842}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:30.883 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:56:30.883 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'6391", 'reported_seq': 6574, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:27.580031+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:56:27.580031+0000', 'last_peered': '2024-09-18T11:56:27.580031+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:56:27.580031+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 6391, 'log_dups_size': 0, 'ondisk_log_size': 6391, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26161152, 'num_objects': 6387, 'num_object_clones': 0, 'num_object_copies': 12774, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3249, 'num_write_kb': 12996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 2851}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:30.883 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:56:30.883 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'6317", 'reported_seq': 6396, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:26.575057+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:56:26.575057+0000', 'last_peered': '2024-09-18T11:56:26.575057+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:56:26.575057+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6317, 'log_dups_size': 0, 'ondisk_log_size': 6317, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25855186, 'num_objects': 6314, 'num_object_clones': 0, 'num_object_copies': 12628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6314, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6315, 'num_write_kb': 25251, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2787}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:30.883 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:56:30.883 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'6410", 'reported_seq': 6493, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:26.563712+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:56:26.563712+0000', 'last_peered': '2024-09-18T11:56:26.563712+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:56:26.563712+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6410, 'log_dups_size': 0, 'ondisk_log_size': 6410, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26232018, 'num_objects': 6406, 'num_object_clones': 0, 'num_object_copies': 12812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6408, 'num_write_kb': 25620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2847}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:30.884 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:56:30.884 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'6377", 'reported_seq': 10941, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:27.617250+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:56:27.617250+0000', 'last_peered': '2024-09-18T11:56:27.617250+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:56:27.617250+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 6377, 'log_dups_size': 0, 'ondisk_log_size': 6377, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26092754, 'num_objects': 6372, 'num_object_clones': 0, 'num_object_copies': 12744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 1271, 'num_objects_unfound': 0, 'num_objects_dirty': 6372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6375, 'num_write_kb': 25484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2211, 'num_bytes_recovered': 9039872, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1271}, {'shards': '2', 'objects': 5100}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:30.884 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:56:30.884 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'6415", 'reported_seq': 6491, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:26.572927+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:56:26.572927+0000', 'last_peered': '2024-09-18T11:56:26.572927+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:56:26.572927+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6415, 'log_dups_size': 0, 'ondisk_log_size': 6415, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26244306, 'num_objects': 6409, 'num_object_clones': 0, 'num_object_copies': 12818, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6413, 'num_write_kb': 25632, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2877}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:30.884 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:56:30.884 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'6449", 'reported_seq': 6503, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:27.628227+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:56:27.628227+0000', 'last_peered': '2024-09-18T11:56:27.628227+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:56:27.628227+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6449, 'log_dups_size': 0, 'ondisk_log_size': 6449, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26395858, 'num_objects': 6446, 'num_object_clones': 0, 'num_object_copies': 12892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6446, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6447, 'num_write_kb': 25779, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 2896}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:30.884 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:56:30.884 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'6422", 'reported_seq': 6629, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:27.577938+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:56:27.577938+0000', 'last_peered': '2024-09-18T11:56:27.577938+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:56:27.577938+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 6422, 'log_dups_size': 0, 'ondisk_log_size': 6422, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26279936, 'num_objects': 6416, 'num_object_clones': 0, 'num_object_copies': 12832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2902, 'num_write_kb': 11608, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 2823}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:30.885 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:56:30.885 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'6292", 'reported_seq': 6372, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:26.540544+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:56:26.540544+0000', 'last_peered': '2024-09-18T11:56:26.540544+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:56:26.540544+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6292, 'log_dups_size': 0, 'ondisk_log_size': 6292, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25764457, 'num_objects': 6291, 'num_object_clones': 0, 'num_object_copies': 12582, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6291, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6291, 'num_write_kb': 25161, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2756}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:30.885 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:56:30.885 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'6436", 'reported_seq': 6518, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:26.662542+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:56:26.662542+0000', 'last_peered': '2024-09-18T11:56:26.662542+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:56:26.662542+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6436, 'log_dups_size': 0, 'ondisk_log_size': 6436, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26330939, 'num_objects': 6431, 'num_object_clones': 0, 'num_object_copies': 12862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6433, 'num_write_kb': 25716, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2909}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:30.885 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:56:30.885 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'6351", 'reported_seq': 6430, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:26.591795+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:56:26.591795+0000', 'last_peered': '2024-09-18T11:56:26.591795+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:56:26.591795+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6351, 'log_dups_size': 0, 'ondisk_log_size': 6351, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25998546, 'num_objects': 6349, 'num_object_clones': 0, 'num_object_copies': 12698, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6349, 'num_write_kb': 25390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2756}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:30.885 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:56:30.885 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'6426", 'reported_seq': 6509, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:26.655640+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:56:26.655640+0000', 'last_peered': '2024-09-18T11:56:26.655640+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:56:26.655640+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6426, 'log_dups_size': 0, 'ondisk_log_size': 6426, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26274212, 'num_objects': 6418, 'num_object_clones': 0, 'num_object_copies': 12836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6423, 'num_write_kb': 25663, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2895}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:30.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:31.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 125 20 104349 104329 3.25934 1.85156 0.0327821 0.0239592 2024-09-18T11:56:31.739 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:31.739 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/204588 objects degraded (19.107%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":191,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":102294,"data_bytes":418918471,"bytes_used":1778995200,"bytes_avail":288131297280,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":204588,"degraded_ratio":0.19106692474631942,"misplaced_objects":1271,"misplaced_total":204588,"misplaced_ratio":0.0062124855807769762,"recovering_objects_per_sec":27,"recovering_bytes_per_sec":112271,"recovering_keys_per_sec":0,"num_objects_recovered":329,"num_bytes_recovered":1347584,"num_keys_recovered":0,"write_bytes_sec":1726392,"read_op_per_sec":0,"write_op_per_sec":421},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:31.797 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:31.797 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:32.127 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 126 20 104841 104821 3.24872 1.92188 0.0884027 0.024038 2024-09-18T11:56:32.521 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:32.537 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:32.565 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-18T11:56:31.070272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":418918471,"num_objects":102294,"num_object_clones":0,"num_object_copies":204588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1271,"num_objects_unfound":0,"num_objects_dirty":102294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86349,"num_write_kb":345270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2232,"num_bytes_recovered":9048700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102357,"ondisk_log_size":102357,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1737300,"kb_used_data":827132,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281378220,"statfs":{"total":289910292480,"available":288131297280,"internally_reserved":0,"allocated":846983168,"data_stored":846266876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,10,17,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20721664,"num_objects":5059,"num_object_clones":0,"num_object_copies":10118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-329,"num_objects_unfound":0,"num_objects_dirty":5059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5059,"num_write_kb":20236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":329,"num_bytes_recovered":1347584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002872"},"pg_stats":[{"pgid":"2.f","version":"37'6441","reported_seq":6510,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.627721+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:26.627721+0000","last_peered":"2024-09-18T11:56:26.627721+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:26.627721+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6441,"log_dups_size":0,"ondisk_log_size":6441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26382336,"num_objects":6441,"num_object_clones":0,"num_object_copies":12882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6441,"num_write_kb":25764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2842},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6420","reported_seq":6576,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:27.604965+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:27.604965+0000","last_peered":"2024-09-18T11:56:27.604965+0000","last_clean":"2024-09-18T11:56:27.604965+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:27.604965+0000","last_undegraded":"2024-09-18T11:56:27.604965+0000","last_fullsized":"2024-09-18T11:56:27.604965+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6420,"log_dups_size":0,"ondisk_log_size":6420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26271744,"num_objects":6414,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3612,"num_write_kb":14448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6391","reported_seq":6574,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:27.580031+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:27.580031+0000","last_peered":"2024-09-18T11:56:27.580031+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:27.580031+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26161152,"num_objects":6387,"num_object_clones":0,"num_object_copies":12774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3249,"num_write_kb":12996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":2851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6377","reported_seq":6404,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:27.607886+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:27.607886+0000","last_peered":"2024-09-18T11:56:27.607886+0000","last_clean":"2024-09-18T11:56:27.607886+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:27.607886+0000","last_undegraded":"2024-09-18T11:56:27.607886+0000","last_fullsized":"2024-09-18T11:56:27.607886+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6377,"log_dups_size":0,"ondisk_log_size":6377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26120192,"num_objects":6377,"num_object_clones":0,"num_object_copies":12754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6377,"num_write_kb":25508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6372","reported_seq":6561,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:26.592089+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:26.592089+0000","last_peered":"2024-09-18T11:56:26.592089+0000","last_clean":"2024-09-18T11:56:26.592089+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:26.592089+0000","last_undegraded":"2024-09-18T11:56:26.592089+0000","last_fullsized":"2024-09-18T11:56:26.592089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6372,"log_dups_size":0,"ondisk_log_size":6372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26071040,"num_objects":6365,"num_object_clones":0,"num_object_copies":12730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6317","reported_seq":6396,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.575057+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:26.575057+0000","last_peered":"2024-09-18T11:56:26.575057+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:26.575057+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6317,"log_dups_size":0,"ondisk_log_size":6317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25855186,"num_objects":6314,"num_object_clones":0,"num_object_copies":12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6315,"num_write_kb":25251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2787},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6410","reported_seq":6493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.563712+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:26.563712+0000","last_peered":"2024-09-18T11:56:26.563712+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:26.563712+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26232018,"num_objects":6406,"num_object_clones":0,"num_object_copies":12812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6408,"num_write_kb":25620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2847},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6458","reported_seq":6681,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:26.644227+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:26.644227+0000","last_peered":"2024-09-18T11:56:26.644227+0000","last_clean":"2024-09-18T11:56:26.644227+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:26.644227+0000","last_undegraded":"2024-09-18T11:56:26.644227+0000","last_fullsized":"2024-09-18T11:56:26.644227+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6458,"log_dups_size":0,"ondisk_log_size":6458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26443776,"num_objects":6456,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2775,"num_write_kb":11100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6377","reported_seq":10941,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:27.617250+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:27.617250+0000","last_peered":"2024-09-18T11:56:27.617250+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:27.617250+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6377,"log_dups_size":0,"ondisk_log_size":6377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26092754,"num_objects":6372,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1271,"num_objects_unfound":0,"num_objects_dirty":6372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6375,"num_write_kb":25484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2211,"num_bytes_recovered":9039872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1271},{"shards":"2","objects":5100}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6415","reported_seq":6491,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.572927+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:26.572927+0000","last_peered":"2024-09-18T11:56:26.572927+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:26.572927+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26244306,"num_objects":6409,"num_object_clones":0,"num_object_copies":12818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6413,"num_write_kb":25632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2877},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6449","reported_seq":6503,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:27.628227+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:27.628227+0000","last_peered":"2024-09-18T11:56:27.628227+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:27.628227+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26395858,"num_objects":6446,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":25779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":2896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6422","reported_seq":6629,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:27.577938+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:27.577938+0000","last_peered":"2024-09-18T11:56:27.577938+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:27.577938+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26279936,"num_objects":6416,"num_object_clones":0,"num_object_copies":12832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2823}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6292","reported_seq":6372,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.540544+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:26.540544+0000","last_peered":"2024-09-18T11:56:26.540544+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:26.540544+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25764457,"num_objects":6291,"num_object_clones":0,"num_object_copies":12582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":25161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2756},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6436","reported_seq":6518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.662542+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:26.662542+0000","last_peered":"2024-09-18T11:56:26.662542+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:26.662542+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26330939,"num_objects":6431,"num_object_clones":0,"num_object_copies":12862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6433,"num_write_kb":25716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2909},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6351","reported_seq":6430,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.591795+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:26.591795+0000","last_peered":"2024-09-18T11:56:26.591795+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:26.591795+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6351,"log_dups_size":0,"ondisk_log_size":6351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25998546,"num_objects":6349,"num_object_clones":0,"num_object_copies":12698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":25390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2756},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6426","reported_seq":6509,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:26.655640+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:26.655640+0000","last_peered":"2024-09-18T11:56:26.655640+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:26.655640+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26274212,"num_objects":6418,"num_object_clones":0,"num_object_copies":12836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6423,"num_write_kb":25663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":418918452,"num_objects":102292,"num_object_clones":0,"num_object_copies":204584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1271,"num_objects_unfound":0,"num_objects_dirty":102292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86347,"num_write_kb":345268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2231,"num_bytes_recovered":9048681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":846569472,"data_stored":846489455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102354,"ondisk_log_size":102354,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051615,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616812,"kb_used_data":318948,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93755028,"statfs":{"total":96636764160,"available":96005148672,"internally_reserved":0,"allocated":326602752,"data_stored":326389596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.439}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.45900000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855378,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455596,"kb_used_data":145252,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93916244,"statfs":{"total":96636764160,"available":96170233856,"internally_reserved":0,"allocated":148738048,"data_stored":148451690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.24099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":4.1340000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.39500000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664892,"kb_used_data":362932,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93706948,"statfs":{"total":96636764160,"available":95955914752,"internally_reserved":0,"allocated":371642368,"data_stored":371425590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,7,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":2.7770000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":371703808,"data_stored":371700329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148480000,"data_stored":148406941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":326385664,"data_stored":326382185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:33.128 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 127 20 105329 105309 3.23815 1.90625 0.0107107 0.0241178 2024-09-18T11:56:34.133 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 128 20 105836 105816 3.22819 1.98047 0.0375355 0.0241917 2024-09-18T11:56:35.133 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 129 20 106346 106326 3.21861 1.99219 0.111004 0.0242645 2024-09-18T11:56:35.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:36.133 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 130 20 106831 106811 3.20843 1.89453 0.0772921 0.02434 2024-09-18T11:56:36.285 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:36.306 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:36.319 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-18T11:56:35.071232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":429224007,"num_objects":104810,"num_object_clones":0,"num_object_copies":209620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1088,"num_objects_unfound":0,"num_objects_dirty":104810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88865,"num_write_kb":355334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2415,"num_bytes_recovered":9798268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104873,"ondisk_log_size":104873,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1757972,"kb_used_data":847804,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281357548,"statfs":{"total":289910292480,"available":288110129152,"internally_reserved":0,"allocated":868151296,"data_stored":867435004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,3,11,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20619264,"num_objects":5034,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-357,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":20136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":357,"num_bytes_recovered":1462272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002889"},"pg_stats":[{"pgid":"2.f","version":"37'6594","reported_seq":6663,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.576405+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:31.576405+0000","last_peered":"2024-09-18T11:56:31.576405+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:31.576405+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6594,"log_dups_size":0,"ondisk_log_size":6594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27009024,"num_objects":6594,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6594,"num_write_kb":26376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2995},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6596","reported_seq":6752,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:32.645543+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:32.645543+0000","last_peered":"2024-09-18T11:56:32.645543+0000","last_clean":"2024-09-18T11:56:32.645543+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:32.645543+0000","last_undegraded":"2024-09-18T11:56:32.645543+0000","last_fullsized":"2024-09-18T11:56:32.645543+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26992640,"num_objects":6590,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3788,"num_write_kb":15152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6544","reported_seq":6727,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:32.629870+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:32.629870+0000","last_peered":"2024-09-18T11:56:32.629870+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:32.629870+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26787840,"num_objects":6540,"num_object_clones":0,"num_object_copies":13080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3402,"num_write_kb":13608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3004}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6509","reported_seq":6536,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:32.578033+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:32.578033+0000","last_peered":"2024-09-18T11:56:32.578033+0000","last_clean":"2024-09-18T11:56:32.578033+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:32.578033+0000","last_undegraded":"2024-09-18T11:56:32.578033+0000","last_fullsized":"2024-09-18T11:56:32.578033+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6509,"log_dups_size":0,"ondisk_log_size":6509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26660864,"num_objects":6509,"num_object_clones":0,"num_object_copies":13018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6509,"num_write_kb":26036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6543","reported_seq":6730,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:31.657243+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:31.657243+0000","last_peered":"2024-09-18T11:56:31.657243+0000","last_clean":"2024-09-18T11:56:31.657243+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:31.657243+0000","last_undegraded":"2024-09-18T11:56:31.657243+0000","last_fullsized":"2024-09-18T11:56:31.657243+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26771456,"num_objects":6536,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3708,"num_write_kb":14832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6472","reported_seq":6551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.621176+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:31.621176+0000","last_peered":"2024-09-18T11:56:31.621176+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:31.621176+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26490066,"num_objects":6469,"num_object_clones":0,"num_object_copies":12938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6470,"num_write_kb":25871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2942},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6567","reported_seq":6652,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.635259+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:31.635259+0000","last_peered":"2024-09-18T11:56:31.635259+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:31.635259+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6567,"log_dups_size":0,"ondisk_log_size":6567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26875090,"num_objects":6563,"num_object_clones":0,"num_object_copies":13126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6565,"num_write_kb":26248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3004},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6636","reported_seq":6859,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:31.651333+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:31.651333+0000","last_peered":"2024-09-18T11:56:31.651333+0000","last_clean":"2024-09-18T11:56:31.651333+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:31.651333+0000","last_undegraded":"2024-09-18T11:56:31.651333+0000","last_fullsized":"2024-09-18T11:56:31.651333+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6636,"log_dups_size":0,"ondisk_log_size":6636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27172864,"num_objects":6634,"num_object_clones":0,"num_object_copies":13268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2953,"num_write_kb":11812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6524","reported_seq":11459,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:32.645432+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:32.645432+0000","last_peered":"2024-09-18T11:56:32.645432+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:32.645432+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6524,"log_dups_size":0,"ondisk_log_size":6524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26694866,"num_objects":6519,"num_object_clones":0,"num_object_copies":13038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1088,"num_objects_unfound":0,"num_objects_dirty":6519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6522,"num_write_kb":26072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2394,"num_bytes_recovered":9789440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1088},{"shards":"2","objects":5430}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6570","reported_seq":6646,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.642561+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:31.642561+0000","last_peered":"2024-09-18T11:56:31.642561+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:31.642561+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6570,"log_dups_size":0,"ondisk_log_size":6570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26879186,"num_objects":6564,"num_object_clones":0,"num_object_copies":13128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6568,"num_write_kb":26252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3032},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6601","reported_seq":6653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:32.594448+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:32.594448+0000","last_peered":"2024-09-18T11:56:32.594448+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:32.594448+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6601,"log_dups_size":0,"ondisk_log_size":6601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27018450,"num_objects":6598,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6599,"num_write_kb":26387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3048}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6571","reported_seq":6778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:32.624150+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:32.624150+0000","last_peered":"2024-09-18T11:56:32.624150+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:32.624150+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6571,"log_dups_size":0,"ondisk_log_size":6571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26890240,"num_objects":6565,"num_object_clones":0,"num_object_copies":13130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3051,"num_write_kb":12204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2972}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6450","reported_seq":6530,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.674917+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:31.674917+0000","last_peered":"2024-09-18T11:56:31.674917+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:31.674917+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411625,"num_objects":6449,"num_object_clones":0,"num_object_copies":12898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6449,"num_write_kb":25793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2914},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6611","reported_seq":6693,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.654523+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:31.654523+0000","last_peered":"2024-09-18T11:56:31.654523+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:31.654523+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27047739,"num_objects":6606,"num_object_clones":0,"num_object_copies":13212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6608,"num_write_kb":26416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3084},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6497","reported_seq":6576,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.618489+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:31.618489+0000","last_peered":"2024-09-18T11:56:31.618489+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:31.618489+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6497,"log_dups_size":0,"ondisk_log_size":6497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26596562,"num_objects":6495,"num_object_clones":0,"num_object_copies":12990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6495,"num_write_kb":25974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2902},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6585","reported_seq":6667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.635340+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:31.635340+0000","last_peered":"2024-09-18T11:56:31.635340+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:31.635340+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6585,"log_dups_size":0,"ondisk_log_size":6585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26925476,"num_objects":6577,"num_object_clones":0,"num_object_copies":13154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6582,"num_write_kb":26299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3054},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":429223988,"num_objects":104808,"num_object_clones":0,"num_object_copies":209616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1088,"num_objects_unfound":0,"num_objects_dirty":104808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88863,"num_write_kb":355332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2414,"num_bytes_recovered":9798249,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":867934208,"data_stored":867854191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104870,"ondisk_log_size":104870,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051616,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625872,"kb_used_data":323848,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93745968,"statfs":{"total":96636764160,"available":95995871232,"internally_reserved":0,"allocated":331620352,"data_stored":331407196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.26400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.42399999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855379,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455420,"kb_used_data":153396,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93916420,"statfs":{"total":96636764160,"available":96170414080,"internally_reserved":0,"allocated":157077504,"data_stored":156791146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":2.8660000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.36299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676680,"kb_used_data":370560,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93695160,"statfs":{"total":96636764160,"available":95943843840,"internally_reserved":0,"allocated":379453440,"data_stored":379236662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,1,11,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":6.3520000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379613184,"data_stored":379609705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156864512,"data_stored":156791453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331456512,"data_stored":331453033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:36.320 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:37.064 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:37.086 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:37.100 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-18T11:56:35.071232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":429224007,"num_objects":104810,"num_object_clones":0,"num_object_copies":209620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1088,"num_objects_unfound":0,"num_objects_dirty":104810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88865,"num_write_kb":355334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2415,"num_bytes_recovered":9798268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104873,"ondisk_log_size":104873,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1757972,"kb_used_data":847804,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281357548,"statfs":{"total":289910292480,"available":288110129152,"internally_reserved":0,"allocated":868151296,"data_stored":867435004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,3,11,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20619264,"num_objects":5034,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-357,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":20136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":357,"num_bytes_recovered":1462272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002889"},"pg_stats":[{"pgid":"2.f","version":"37'6594","reported_seq":6663,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.576405+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:31.576405+0000","last_peered":"2024-09-18T11:56:31.576405+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:31.576405+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6594,"log_dups_size":0,"ondisk_log_size":6594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27009024,"num_objects":6594,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6594,"num_write_kb":26376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2995},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6596","reported_seq":6752,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:32.645543+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:32.645543+0000","last_peered":"2024-09-18T11:56:32.645543+0000","last_clean":"2024-09-18T11:56:32.645543+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:32.645543+0000","last_undegraded":"2024-09-18T11:56:32.645543+0000","last_fullsized":"2024-09-18T11:56:32.645543+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26992640,"num_objects":6590,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3788,"num_write_kb":15152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6544","reported_seq":6727,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:32.629870+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:32.629870+0000","last_peered":"2024-09-18T11:56:32.629870+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:32.629870+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26787840,"num_objects":6540,"num_object_clones":0,"num_object_copies":13080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3402,"num_write_kb":13608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3004}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6509","reported_seq":6536,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:32.578033+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:32.578033+0000","last_peered":"2024-09-18T11:56:32.578033+0000","last_clean":"2024-09-18T11:56:32.578033+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:32.578033+0000","last_undegraded":"2024-09-18T11:56:32.578033+0000","last_fullsized":"2024-09-18T11:56:32.578033+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6509,"log_dups_size":0,"ondisk_log_size":6509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26660864,"num_objects":6509,"num_object_clones":0,"num_object_copies":13018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6509,"num_write_kb":26036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6543","reported_seq":6730,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:31.657243+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:31.657243+0000","last_peered":"2024-09-18T11:56:31.657243+0000","last_clean":"2024-09-18T11:56:31.657243+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:31.657243+0000","last_undegraded":"2024-09-18T11:56:31.657243+0000","last_fullsized":"2024-09-18T11:56:31.657243+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26771456,"num_objects":6536,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3708,"num_write_kb":14832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6472","reported_seq":6551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.621176+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:31.621176+0000","last_peered":"2024-09-18T11:56:31.621176+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:31.621176+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26490066,"num_objects":6469,"num_object_clones":0,"num_object_copies":12938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6470,"num_write_kb":25871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2942},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6567","reported_seq":6652,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.635259+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:31.635259+0000","last_peered":"2024-09-18T11:56:31.635259+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:31.635259+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6567,"log_dups_size":0,"ondisk_log_size":6567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26875090,"num_objects":6563,"num_object_clones":0,"num_object_copies":13126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6565,"num_write_kb":26248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3004},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6636","reported_seq":6859,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:31.651333+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:31.651333+0000","last_peered":"2024-09-18T11:56:31.651333+0000","last_clean":"2024-09-18T11:56:31.651333+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:31.651333+0000","last_undegraded":"2024-09-18T11:56:31.651333+0000","last_fullsized":"2024-09-18T11:56:31.651333+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6636,"log_dups_size":0,"ondisk_log_size":6636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27172864,"num_objects":6634,"num_object_clones":0,"num_object_copies":13268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2953,"num_write_kb":11812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6524","reported_seq":11459,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:32.645432+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:32.645432+0000","last_peered":"2024-09-18T11:56:32.645432+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:32.645432+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6524,"log_dups_size":0,"ondisk_log_size":6524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26694866,"num_objects":6519,"num_object_clones":0,"num_object_copies":13038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1088,"num_objects_unfound":0,"num_objects_dirty":6519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6522,"num_write_kb":26072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2394,"num_bytes_recovered":9789440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1088},{"shards":"2","objects":5430}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6570","reported_seq":6646,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.642561+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:31.642561+0000","last_peered":"2024-09-18T11:56:31.642561+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:31.642561+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6570,"log_dups_size":0,"ondisk_log_size":6570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26879186,"num_objects":6564,"num_object_clones":0,"num_object_copies":13128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6568,"num_write_kb":26252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3032},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6601","reported_seq":6653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:32.594448+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:32.594448+0000","last_peered":"2024-09-18T11:56:32.594448+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:32.594448+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6601,"log_dups_size":0,"ondisk_log_size":6601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27018450,"num_objects":6598,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6599,"num_write_kb":26387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3048}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6571","reported_seq":6778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:32.624150+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:32.624150+0000","last_peered":"2024-09-18T11:56:32.624150+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:32.624150+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6571,"log_dups_size":0,"ondisk_log_size":6571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26890240,"num_objects":6565,"num_object_clones":0,"num_object_copies":13130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3051,"num_write_kb":12204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2972}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6450","reported_seq":6530,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.674917+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:31.674917+0000","last_peered":"2024-09-18T11:56:31.674917+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:31.674917+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411625,"num_objects":6449,"num_object_clones":0,"num_object_copies":12898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6449,"num_write_kb":25793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2914},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6611","reported_seq":6693,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.654523+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:31.654523+0000","last_peered":"2024-09-18T11:56:31.654523+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:31.654523+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27047739,"num_objects":6606,"num_object_clones":0,"num_object_copies":13212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6608,"num_write_kb":26416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3084},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6497","reported_seq":6576,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.618489+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:31.618489+0000","last_peered":"2024-09-18T11:56:31.618489+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:31.618489+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6497,"log_dups_size":0,"ondisk_log_size":6497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26596562,"num_objects":6495,"num_object_clones":0,"num_object_copies":12990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6495,"num_write_kb":25974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2902},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6585","reported_seq":6667,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:31.635340+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:31.635340+0000","last_peered":"2024-09-18T11:56:31.635340+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:31.635340+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6585,"log_dups_size":0,"ondisk_log_size":6585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26925476,"num_objects":6577,"num_object_clones":0,"num_object_copies":13154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6582,"num_write_kb":26299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3054},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":429223988,"num_objects":104808,"num_object_clones":0,"num_object_copies":209616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1088,"num_objects_unfound":0,"num_objects_dirty":104808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88863,"num_write_kb":355332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2414,"num_bytes_recovered":9798249,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":867934208,"data_stored":867854191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104870,"ondisk_log_size":104870,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051616,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625872,"kb_used_data":323848,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93745968,"statfs":{"total":96636764160,"available":95995871232,"internally_reserved":0,"allocated":331620352,"data_stored":331407196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.26400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.42399999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855379,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455420,"kb_used_data":153396,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93916420,"statfs":{"total":96636764160,"available":96170414080,"internally_reserved":0,"allocated":157077504,"data_stored":156791146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":2.8660000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.36299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676680,"kb_used_data":370560,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93695160,"statfs":{"total":96636764160,"available":95943843840,"internally_reserved":0,"allocated":379453440,"data_stored":379236662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,1,11,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":6.3520000000000003}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379613184,"data_stored":379609705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156864512,"data_stored":156791453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331456512,"data_stored":331453033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:37.101 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:56:37.101 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'6594", 'reported_seq': 6663, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:31.576405+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:56:31.576405+0000', 'last_peered': '2024-09-18T11:56:31.576405+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:56:31.576405+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6594, 'log_dups_size': 0, 'ondisk_log_size': 6594, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27009024, 'num_objects': 6594, 'num_object_clones': 0, 'num_object_copies': 13188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6594, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6594, 'num_write_kb': 26376, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2995}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:37.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:56:37.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'6544", 'reported_seq': 6727, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:32.629870+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:56:32.629870+0000', 'last_peered': '2024-09-18T11:56:32.629870+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:56:32.629870+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 6544, 'log_dups_size': 0, 'ondisk_log_size': 6544, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26787840, 'num_objects': 6540, 'num_object_clones': 0, 'num_object_copies': 13080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3402, 'num_write_kb': 13608, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 3004}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:37.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:56:37.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'6472", 'reported_seq': 6551, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:31.621176+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:56:31.621176+0000', 'last_peered': '2024-09-18T11:56:31.621176+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:56:31.621176+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6472, 'log_dups_size': 0, 'ondisk_log_size': 6472, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26490066, 'num_objects': 6469, 'num_object_clones': 0, 'num_object_copies': 12938, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6470, 'num_write_kb': 25871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2942}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:37.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:56:37.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'6567", 'reported_seq': 6652, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:31.635259+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:56:31.635259+0000', 'last_peered': '2024-09-18T11:56:31.635259+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:56:31.635259+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6567, 'log_dups_size': 0, 'ondisk_log_size': 6567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26875090, 'num_objects': 6563, 'num_object_clones': 0, 'num_object_copies': 13126, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6565, 'num_write_kb': 26248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3004}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:37.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:56:37.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'6524", 'reported_seq': 11459, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:32.645432+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:56:32.645432+0000', 'last_peered': '2024-09-18T11:56:32.645432+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:56:32.645432+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 6524, 'log_dups_size': 0, 'ondisk_log_size': 6524, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26694866, 'num_objects': 6519, 'num_object_clones': 0, 'num_object_copies': 13038, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 1088, 'num_objects_unfound': 0, 'num_objects_dirty': 6519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6522, 'num_write_kb': 26072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2394, 'num_bytes_recovered': 9789440, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1088}, {'shards': '2', 'objects': 5430}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:37.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:56:37.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'6570", 'reported_seq': 6646, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:31.642561+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:56:31.642561+0000', 'last_peered': '2024-09-18T11:56:31.642561+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:56:31.642561+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6570, 'log_dups_size': 0, 'ondisk_log_size': 6570, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26879186, 'num_objects': 6564, 'num_object_clones': 0, 'num_object_copies': 13128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6568, 'num_write_kb': 26252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3032}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:37.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:56:37.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'6601", 'reported_seq': 6653, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:32.594448+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:56:32.594448+0000', 'last_peered': '2024-09-18T11:56:32.594448+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:56:32.594448+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6601, 'log_dups_size': 0, 'ondisk_log_size': 6601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27018450, 'num_objects': 6598, 'num_object_clones': 0, 'num_object_copies': 13196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6599, 'num_write_kb': 26387, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 3048}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:37.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:56:37.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'6571", 'reported_seq': 6778, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:32.624150+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:56:32.624150+0000', 'last_peered': '2024-09-18T11:56:32.624150+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:56:32.624150+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 6571, 'log_dups_size': 0, 'ondisk_log_size': 6571, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26890240, 'num_objects': 6565, 'num_object_clones': 0, 'num_object_copies': 13130, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6565, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3051, 'num_write_kb': 12204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 2972}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:37.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:56:37.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'6450", 'reported_seq': 6530, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:31.674917+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:56:31.674917+0000', 'last_peered': '2024-09-18T11:56:31.674917+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:56:31.674917+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6450, 'log_dups_size': 0, 'ondisk_log_size': 6450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26411625, 'num_objects': 6449, 'num_object_clones': 0, 'num_object_copies': 12898, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6449, 'num_write_kb': 25793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2914}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:37.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:56:37.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'6611", 'reported_seq': 6693, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:31.654523+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:56:31.654523+0000', 'last_peered': '2024-09-18T11:56:31.654523+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:56:31.654523+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6611, 'log_dups_size': 0, 'ondisk_log_size': 6611, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27047739, 'num_objects': 6606, 'num_object_clones': 0, 'num_object_copies': 13212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6608, 'num_write_kb': 26416, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3084}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:37.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:56:37.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'6497", 'reported_seq': 6576, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:31.618489+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:56:31.618489+0000', 'last_peered': '2024-09-18T11:56:31.618489+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:56:31.618489+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6497, 'log_dups_size': 0, 'ondisk_log_size': 6497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26596562, 'num_objects': 6495, 'num_object_clones': 0, 'num_object_copies': 12990, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6495, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6495, 'num_write_kb': 25974, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2902}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:37.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:56:37.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'6585", 'reported_seq': 6667, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:31.635340+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:56:31.635340+0000', 'last_peered': '2024-09-18T11:56:31.635340+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:56:31.635340+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6585, 'log_dups_size': 0, 'ondisk_log_size': 6585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26925476, 'num_objects': 6577, 'num_object_clones': 0, 'num_object_copies': 13154, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6582, 'num_write_kb': 26299, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3054}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:37.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:37.133 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 131 20 107308 107288 3.19816 1.86328 0.0204424 0.024419 2024-09-18T11:56:38.032 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:38.032 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/212780 objects degraded (18.371%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":198,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":106390,"data_bytes":435695687,"bytes_used":1812201472,"bytes_avail":288098091008,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":212780,"degraded_ratio":0.18371087508224457,"misplaced_objects":1088,"misplaced_total":212780,"misplaced_ratio":0.005113262524673372,"recovering_objects_per_sec":29,"recovering_bytes_per_sec":121827,"recovering_keys_per_sec":0,"num_objects_recovered":357,"num_bytes_recovered":1462272,"num_keys_recovered":0,"write_bytes_sec":2257043,"read_op_per_sec":0,"write_op_per_sec":551},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:38.059 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:38.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:38.133 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 132 20 107775 107755 3.18775 1.82422 0.0402798 0.0244985 2024-09-18T11:56:38.837 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:38.854 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:38.865 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-18T11:56:37.071725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":435695687,"num_objects":106390,"num_object_clones":0,"num_object_copies":212780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1088,"num_objects_unfound":0,"num_objects_dirty":106390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90445,"num_write_kb":361654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2415,"num_bytes_recovered":9798268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106453,"ondisk_log_size":106453,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1769728,"kb_used_data":863720,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281345792,"statfs":{"total":289910292480,"available":288098091008,"internally_reserved":0,"allocated":884449280,"data_stored":883732988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,4,4,9,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27090944,"num_objects":6614,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-357,"num_objects_unfound":0,"num_objects_dirty":6614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6614,"num_write_kb":26456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":357,"num_bytes_recovered":1462272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002844"},"pg_stats":[{"pgid":"2.f","version":"37'6756","reported_seq":6823,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.639686+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:36.639686+0000","last_peered":"2024-09-18T11:56:36.639686+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:36.639686+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6756,"log_dups_size":0,"ondisk_log_size":6756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27672576,"num_objects":6756,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":27024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3157},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6596","reported_seq":6752,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:32.645543+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:32.645543+0000","last_peered":"2024-09-18T11:56:32.645543+0000","last_clean":"2024-09-18T11:56:32.645543+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:32.645543+0000","last_undegraded":"2024-09-18T11:56:32.645543+0000","last_fullsized":"2024-09-18T11:56:32.645543+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26992640,"num_objects":6590,"num_object_clones":0,"num_object_copies":13180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3788,"num_write_kb":15152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6544","reported_seq":6727,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:32.629870+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:32.629870+0000","last_peered":"2024-09-18T11:56:32.629870+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:32.629870+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26787840,"num_objects":6540,"num_object_clones":0,"num_object_copies":13080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3402,"num_write_kb":13608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3004}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6509","reported_seq":6536,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:32.578033+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:32.578033+0000","last_peered":"2024-09-18T11:56:32.578033+0000","last_clean":"2024-09-18T11:56:32.578033+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:32.578033+0000","last_undegraded":"2024-09-18T11:56:32.578033+0000","last_fullsized":"2024-09-18T11:56:32.578033+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6509,"log_dups_size":0,"ondisk_log_size":6509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26660864,"num_objects":6509,"num_object_clones":0,"num_object_copies":13018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6509,"num_write_kb":26036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6695","reported_seq":6884,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:36.686629+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:36.686629+0000","last_peered":"2024-09-18T11:56:36.686629+0000","last_clean":"2024-09-18T11:56:36.686629+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:36.686629+0000","last_undegraded":"2024-09-18T11:56:36.686629+0000","last_fullsized":"2024-09-18T11:56:36.686629+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6695,"log_dups_size":0,"ondisk_log_size":6695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3860,"num_write_kb":15440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6642","reported_seq":6721,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.666760+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:36.666760+0000","last_peered":"2024-09-18T11:56:36.666760+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:36.666760+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6642,"log_dups_size":0,"ondisk_log_size":6642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27186386,"num_objects":6639,"num_object_clones":0,"num_object_copies":13278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6640,"num_write_kb":26551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3112},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6720","reported_seq":6805,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.524663+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:36.524663+0000","last_peered":"2024-09-18T11:56:36.524663+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:36.524663+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6720,"log_dups_size":0,"ondisk_log_size":6720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27501778,"num_objects":6716,"num_object_clones":0,"num_object_copies":13432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6718,"num_write_kb":26860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3157},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6775","reported_seq":6998,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:36.629267+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:36.629267+0000","last_peered":"2024-09-18T11:56:36.629267+0000","last_clean":"2024-09-18T11:56:36.629267+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:36.629267+0000","last_undegraded":"2024-09-18T11:56:36.629267+0000","last_fullsized":"2024-09-18T11:56:36.629267+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6775,"log_dups_size":0,"ondisk_log_size":6775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27742208,"num_objects":6773,"num_object_clones":0,"num_object_copies":13546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":12368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6524","reported_seq":11459,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:32.645432+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:32.645432+0000","last_peered":"2024-09-18T11:56:32.645432+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:32.645432+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6524,"log_dups_size":0,"ondisk_log_size":6524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26694866,"num_objects":6519,"num_object_clones":0,"num_object_copies":13038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1088,"num_objects_unfound":0,"num_objects_dirty":6519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6522,"num_write_kb":26072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2394,"num_bytes_recovered":9789440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1088},{"shards":"2","objects":5430}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6726","reported_seq":6802,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.623571+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:36.623571+0000","last_peered":"2024-09-18T11:56:36.623571+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:36.623571+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27518162,"num_objects":6720,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3188},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6601","reported_seq":6653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:32.594448+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:32.594448+0000","last_peered":"2024-09-18T11:56:32.594448+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:32.594448+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6601,"log_dups_size":0,"ondisk_log_size":6601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27018450,"num_objects":6598,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6599,"num_write_kb":26387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3048}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6571","reported_seq":6778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:32.624150+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:32.624150+0000","last_peered":"2024-09-18T11:56:32.624150+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:32.624150+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6571,"log_dups_size":0,"ondisk_log_size":6571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26890240,"num_objects":6565,"num_object_clones":0,"num_object_copies":13130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3051,"num_write_kb":12204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":2972}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6622","reported_seq":6700,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.707560+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:36.707560+0000","last_peered":"2024-09-18T11:56:36.707560+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:36.707560+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6622,"log_dups_size":0,"ondisk_log_size":6622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116137,"num_objects":6621,"num_object_clones":0,"num_object_copies":13242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6621,"num_write_kb":26481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3086},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6755","reported_seq":6837,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.642436+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:36.642436+0000","last_peered":"2024-09-18T11:56:36.642436+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:36.642436+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6755,"log_dups_size":0,"ondisk_log_size":6755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27637563,"num_objects":6750,"num_object_clones":0,"num_object_copies":13500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6752,"num_write_kb":26992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3228},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6663","reported_seq":6742,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.517179+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:36.517179+0000","last_peered":"2024-09-18T11:56:36.517179+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:36.517179+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27276498,"num_objects":6661,"num_object_clones":0,"num_object_copies":13322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6661,"num_write_kb":26638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3068},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6751","reported_seq":6833,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.682934+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:36.682934+0000","last_peered":"2024-09-18T11:56:36.682934+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:36.682934+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6751,"log_dups_size":0,"ondisk_log_size":6751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27605412,"num_objects":6743,"num_object_clones":0,"num_object_copies":13486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6748,"num_write_kb":26963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3220},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":435695668,"num_objects":106388,"num_object_clones":0,"num_object_copies":212776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":1088,"num_objects_unfound":0,"num_objects_dirty":106388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90443,"num_write_kb":361652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2414,"num_bytes_recovered":9798249,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":884183040,"data_stored":884103023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106450,"ondisk_log_size":106450,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051616,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625872,"kb_used_data":323848,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93745968,"statfs":{"total":96636764160,"available":95995871232,"internally_reserved":0,"allocated":331620352,"data_stored":331407196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.26400000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.42399999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855380,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468044,"kb_used_data":161860,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93903796,"statfs":{"total":96636764160,"available":96157487104,"internally_reserved":0,"allocated":165744640,"data_stored":165458282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.309}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.29299999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675812,"kb_used_data":378012,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93696028,"statfs":{"total":96636764160,"available":95944732672,"internally_reserved":0,"allocated":387084288,"data_stored":386867510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.59799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":3.3090000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.30199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":387215360,"data_stored":387211881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165511168,"data_stored":165438109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331456512,"data_stored":331453033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:39.136 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 133 20 108251 108231 3.17777 1.85938 0.0316971 0.0245772 2024-09-18T11:56:40.133 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 134 20 108763 108743 3.16898 2 0.00648199 0.0246432 2024-09-18T11:56:41.134 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 135 20 109275 109255 3.16032 2 0.0147978 0.0247135 2024-09-18T11:56:41.867 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:42.134 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 136 20 109773 109753 3.15139 1.94531 0.0254566 0.0247794 2024-09-18T11:56:42.563 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:42.579 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:42.596 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-18T11:56:41.072675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":439332935,"num_objects":107278,"num_object_clones":0,"num_object_copies":214556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":893,"num_objects_unfound":0,"num_objects_dirty":107278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91333,"num_write_kb":365206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2610,"num_bytes_recovered":10596988,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107341,"ondisk_log_size":107341,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1778584,"kb_used_data":868416,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281336936,"statfs":{"total":289910292480,"available":288089022464,"internally_reserved":0,"allocated":889257984,"data_stored":888541692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,5,5,10,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20414464,"num_objects":4984,"num_object_clones":0,"num_object_copies":9968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-378,"num_objects_unfound":0,"num_objects_dirty":4984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4984,"num_write_kb":19936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":1548288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002856"},"pg_stats":[{"pgid":"2.f","version":"37'6756","reported_seq":6823,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.639686+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:36.639686+0000","last_peered":"2024-09-18T11:56:36.639686+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:36.639686+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6756,"log_dups_size":0,"ondisk_log_size":6756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27672576,"num_objects":6756,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":27024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3157},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6732","reported_seq":6888,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:37.536164+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:37.536164+0000","last_peered":"2024-09-18T11:56:37.536164+0000","last_clean":"2024-09-18T11:56:37.536164+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:37.536164+0000","last_undegraded":"2024-09-18T11:56:37.536164+0000","last_fullsized":"2024-09-18T11:56:37.536164+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6732,"log_dups_size":0,"ondisk_log_size":6732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27549696,"num_objects":6726,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3924,"num_write_kb":15696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6715","reported_seq":6892,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:37.646118+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:37.646118+0000","last_peered":"2024-09-18T11:56:37.646118+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:37.646118+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6715,"log_dups_size":0,"ondisk_log_size":6715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27488256,"num_objects":6711,"num_object_clones":0,"num_object_copies":13422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3573,"num_write_kb":14292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3175}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6642","reported_seq":6669,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:37.565809+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:37.565809+0000","last_peered":"2024-09-18T11:56:37.565809+0000","last_clean":"2024-09-18T11:56:37.565809+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:37.565809+0000","last_undegraded":"2024-09-18T11:56:37.565809+0000","last_fullsized":"2024-09-18T11:56:37.565809+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6642,"log_dups_size":0,"ondisk_log_size":6642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27205632,"num_objects":6642,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":26568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6695","reported_seq":6884,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:36.686629+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:36.686629+0000","last_peered":"2024-09-18T11:56:36.686629+0000","last_clean":"2024-09-18T11:56:36.686629+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:36.686629+0000","last_undegraded":"2024-09-18T11:56:36.686629+0000","last_fullsized":"2024-09-18T11:56:36.686629+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6695,"log_dups_size":0,"ondisk_log_size":6695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3860,"num_write_kb":15440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6642","reported_seq":6721,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.666760+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:36.666760+0000","last_peered":"2024-09-18T11:56:36.666760+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:36.666760+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6642,"log_dups_size":0,"ondisk_log_size":6642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27186386,"num_objects":6639,"num_object_clones":0,"num_object_copies":13278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6640,"num_write_kb":26551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3112},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6720","reported_seq":6805,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.524663+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:36.524663+0000","last_peered":"2024-09-18T11:56:36.524663+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:36.524663+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6720,"log_dups_size":0,"ondisk_log_size":6720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27501778,"num_objects":6716,"num_object_clones":0,"num_object_copies":13432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6718,"num_write_kb":26860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3157},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6775","reported_seq":6998,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:36.629267+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:36.629267+0000","last_peered":"2024-09-18T11:56:36.629267+0000","last_clean":"2024-09-18T11:56:36.629267+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:36.629267+0000","last_undegraded":"2024-09-18T11:56:36.629267+0000","last_fullsized":"2024-09-18T11:56:36.629267+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6775,"log_dups_size":0,"ondisk_log_size":6775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27742208,"num_objects":6773,"num_object_clones":0,"num_object_copies":13546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":12368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6677","reported_seq":12003,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:37.656493+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:37.656493+0000","last_peered":"2024-09-18T11:56:37.656493+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:37.656493+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6677,"log_dups_size":0,"ondisk_log_size":6677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27321554,"num_objects":6672,"num_object_clones":0,"num_object_copies":13344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":893,"num_objects_unfound":0,"num_objects_dirty":6672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":26684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2589,"num_bytes_recovered":10588160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":893},{"shards":"2","objects":5778}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6726","reported_seq":6802,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.623571+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:36.623571+0000","last_peered":"2024-09-18T11:56:36.623571+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:36.623571+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27518162,"num_objects":6720,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3188},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6758","reported_seq":6812,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:37.626511+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:37.626511+0000","last_peered":"2024-09-18T11:56:37.626511+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:37.626511+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6758,"log_dups_size":0,"ondisk_log_size":6758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27661522,"num_objects":6755,"num_object_clones":0,"num_object_copies":13510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":27015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6709","reported_seq":6916,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:37.580591+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:37.580591+0000","last_peered":"2024-09-18T11:56:37.580591+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:37.580591+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6709,"log_dups_size":0,"ondisk_log_size":6709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27455488,"num_objects":6703,"num_object_clones":0,"num_object_copies":13406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3189,"num_write_kb":12756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3110}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6622","reported_seq":6700,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.707560+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:36.707560+0000","last_peered":"2024-09-18T11:56:36.707560+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:36.707560+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6622,"log_dups_size":0,"ondisk_log_size":6622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27116137,"num_objects":6621,"num_object_clones":0,"num_object_copies":13242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6621,"num_write_kb":26481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3086},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6755","reported_seq":6837,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.642436+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:36.642436+0000","last_peered":"2024-09-18T11:56:36.642436+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:36.642436+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6755,"log_dups_size":0,"ondisk_log_size":6755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27637563,"num_objects":6750,"num_object_clones":0,"num_object_copies":13500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6752,"num_write_kb":26992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3228},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6663","reported_seq":6742,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.517179+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:36.517179+0000","last_peered":"2024-09-18T11:56:36.517179+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:36.517179+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27276498,"num_objects":6661,"num_object_clones":0,"num_object_copies":13322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6661,"num_write_kb":26638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3068},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6751","reported_seq":6833,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:36.682934+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:36.682934+0000","last_peered":"2024-09-18T11:56:36.682934+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:36.682934+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6751,"log_dups_size":0,"ondisk_log_size":6751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27605412,"num_objects":6743,"num_object_clones":0,"num_object_copies":13486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6748,"num_write_kb":26963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3220},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":439332916,"num_objects":107276,"num_object_clones":0,"num_object_copies":214552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":893,"num_objects_unfound":0,"num_objects_dirty":107276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91331,"num_write_kb":365204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2609,"num_bytes_recovered":10596969,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":889085952,"data_stored":889005935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107338,"ondisk_log_size":107338,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051617,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634728,"kb_used_data":328544,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93737112,"statfs":{"total":96636764160,"available":95986802688,"internally_reserved":0,"allocated":336429056,"data_stored":336215900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.318}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":1.018},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.42599999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855380,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468044,"kb_used_data":161860,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93903796,"statfs":{"total":96636764160,"available":96157487104,"internally_reserved":0,"allocated":165744640,"data_stored":165458282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.309}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.29299999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675812,"kb_used_data":378012,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93696028,"statfs":{"total":96636764160,"available":95944732672,"internally_reserved":0,"allocated":387084288,"data_stored":386867510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.59799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":3.3090000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.30199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":387215360,"data_stored":387211881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165511168,"data_stored":165438109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":336359424,"data_stored":336355945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:42.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:43.134 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 137 20 110249 110229 3.14196 1.85938 0.0723429 0.0248563 2024-09-18T11:56:43.373 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:43.378 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:43.400 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-18T11:56:43.073117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":449589319,"num_objects":109782,"num_object_clones":0,"num_object_copies":219564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":689,"num_objects_unfound":0,"num_objects_dirty":109782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93837,"num_write_kb":375222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2814,"num_bytes_recovered":11432572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109845,"ondisk_log_size":109845,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1799324,"kb_used_data":889156,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281316196,"statfs":{"total":289910292480,"available":288067784704,"internally_reserved":0,"allocated":910495744,"data_stored":909779452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,6,7,10,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30670848,"num_objects":7488,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-582,"num_objects_unfound":0,"num_objects_dirty":7488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":29952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":582,"num_bytes_recovered":2383872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002844"},"pg_stats":[{"pgid":"2.f","version":"37'6910","reported_seq":6979,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.709971+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:41.709971+0000","last_peered":"2024-09-18T11:56:41.709971+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:41.709971+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6910,"log_dups_size":0,"ondisk_log_size":6910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28303360,"num_objects":6910,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":27640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3311},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6895","reported_seq":7051,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:42.626646+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:42.626646+0000","last_peered":"2024-09-18T11:56:42.626646+0000","last_clean":"2024-09-18T11:56:42.626646+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:42.626646+0000","last_undegraded":"2024-09-18T11:56:42.626646+0000","last_fullsized":"2024-09-18T11:56:42.626646+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28217344,"num_objects":6889,"num_object_clones":0,"num_object_copies":13778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":16348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6879","reported_seq":7062,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:42.660210+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:42.660210+0000","last_peered":"2024-09-18T11:56:42.660210+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:42.660210+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6879,"log_dups_size":0,"ondisk_log_size":6879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28160000,"num_objects":6875,"num_object_clones":0,"num_object_copies":13750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":14948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3339}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6801","reported_seq":6828,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:42.524225+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:42.524225+0000","last_peered":"2024-09-18T11:56:42.524225+0000","last_clean":"2024-09-18T11:56:42.524225+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:42.524225+0000","last_undegraded":"2024-09-18T11:56:42.524225+0000","last_fullsized":"2024-09-18T11:56:42.524225+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6801,"log_dups_size":0,"ondisk_log_size":6801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27856896,"num_objects":6801,"num_object_clones":0,"num_object_copies":13602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6801,"num_write_kb":27204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6866","reported_seq":7055,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:41.707867+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:41.707867+0000","last_peered":"2024-09-18T11:56:41.707867+0000","last_clean":"2024-09-18T11:56:41.707867+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:41.707867+0000","last_undegraded":"2024-09-18T11:56:41.707867+0000","last_fullsized":"2024-09-18T11:56:41.707867+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6866,"log_dups_size":0,"ondisk_log_size":6866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28094464,"num_objects":6859,"num_object_clones":0,"num_object_copies":13718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4031,"num_write_kb":16124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6786","reported_seq":6865,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.708090+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:41.708090+0000","last_peered":"2024-09-18T11:56:41.708090+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:41.708090+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6786,"log_dups_size":0,"ondisk_log_size":6786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27776210,"num_objects":6783,"num_object_clones":0,"num_object_copies":13566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6784,"num_write_kb":27127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3256},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6869","reported_seq":6953,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.687575+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:41.687575+0000","last_peered":"2024-09-18T11:56:41.687575+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:41.687575+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6869,"log_dups_size":0,"ondisk_log_size":6869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28112082,"num_objects":6865,"num_object_clones":0,"num_object_copies":13730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":27456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3306},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6923","reported_seq":7146,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:41.766497+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:41.766497+0000","last_peered":"2024-09-18T11:56:41.766497+0000","last_clean":"2024-09-18T11:56:41.766497+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:41.766497+0000","last_undegraded":"2024-09-18T11:56:41.766497+0000","last_fullsized":"2024-09-18T11:56:41.766497+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6923,"log_dups_size":0,"ondisk_log_size":6923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28348416,"num_objects":6921,"num_object_clones":0,"num_object_copies":13842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3240,"num_write_kb":12960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6842","reported_seq":12574,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:42.666370+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:42.666370+0000","last_peered":"2024-09-18T11:56:42.666370+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:42.666370+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6842,"log_dups_size":0,"ondisk_log_size":6842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27997394,"num_objects":6837,"num_object_clones":0,"num_object_copies":13674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":689,"num_objects_unfound":0,"num_objects_dirty":6837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6840,"num_write_kb":27344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2793,"num_bytes_recovered":11423744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":689},{"shards":"2","objects":6147}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6876","reported_seq":6952,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.731801+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:41.731801+0000","last_peered":"2024-09-18T11:56:41.731801+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:41.731801+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6876,"log_dups_size":0,"ondisk_log_size":6876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28132562,"num_objects":6870,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6874,"num_write_kb":27476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3338},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6910","reported_seq":6964,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:42.643099+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:42.643099+0000","last_peered":"2024-09-18T11:56:42.643099+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:42.643099+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6910,"log_dups_size":0,"ondisk_log_size":6910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28284114,"num_objects":6907,"num_object_clones":0,"num_object_copies":13814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":27623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3357}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6859","reported_seq":7066,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:42.612892+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:42.612892+0000","last_peered":"2024-09-18T11:56:42.612892+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:42.612892+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6859,"log_dups_size":0,"ondisk_log_size":6859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28069888,"num_objects":6853,"num_object_clones":0,"num_object_copies":13706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3339,"num_write_kb":13356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6800","reported_seq":6880,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.783159+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:41.783159+0000","last_peered":"2024-09-18T11:56:41.783159+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:41.783159+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6800,"log_dups_size":0,"ondisk_log_size":6800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27845225,"num_objects":6799,"num_object_clones":0,"num_object_copies":13598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6799,"num_write_kb":27193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3264},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6925","reported_seq":7007,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.742067+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:41.742067+0000","last_peered":"2024-09-18T11:56:41.742067+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:41.742067+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6925,"log_dups_size":0,"ondisk_log_size":6925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28333883,"num_objects":6920,"num_object_clones":0,"num_object_copies":13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6922,"num_write_kb":27672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3398},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6793","reported_seq":6872,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.752317+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:41.752317+0000","last_peered":"2024-09-18T11:56:41.752317+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:41.752317+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6793,"log_dups_size":0,"ondisk_log_size":6793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27808978,"num_objects":6791,"num_object_clones":0,"num_object_copies":13582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6791,"num_write_kb":27158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3198},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6908","reported_seq":6990,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.771132+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:41.771132+0000","last_peered":"2024-09-18T11:56:41.771132+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:41.771132+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6908,"log_dups_size":0,"ondisk_log_size":6908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28248484,"num_objects":6900,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6905,"num_write_kb":27591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3377},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":449589300,"num_objects":109780,"num_object_clones":0,"num_object_copies":219560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":689,"num_objects_unfound":0,"num_objects_dirty":109780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93835,"num_write_kb":375220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2813,"num_bytes_recovered":11432553,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":910360576,"data_stored":910280559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109842,"ondisk_log_size":109842,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051618,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644032,"kb_used_data":333688,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93727808,"statfs":{"total":96636764160,"available":95977275392,"internally_reserved":0,"allocated":341696512,"data_stored":341483356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.318}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":1.018},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.42599999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855381,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468008,"kb_used_data":170144,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93903832,"statfs":{"total":96636764160,"available":96157523968,"internally_reserved":0,"allocated":174227456,"data_stored":173941098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.253}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":2.7080000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.27000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687284,"kb_used_data":385324,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93684556,"statfs":{"total":96636764160,"available":95932985344,"internally_reserved":0,"allocated":394571776,"data_stored":394354998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":4.4109999999999996},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":4.3319999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":4.46}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":394817536,"data_stored":394814057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173981696,"data_stored":173908637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341561344,"data_stored":341557865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:43.402 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:56:43.402 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'6910", 'reported_seq': 6979, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:41.709971+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:56:41.709971+0000', 'last_peered': '2024-09-18T11:56:41.709971+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:56:41.709971+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6910, 'log_dups_size': 0, 'ondisk_log_size': 6910, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28303360, 'num_objects': 6910, 'num_object_clones': 0, 'num_object_copies': 13820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6910, 'num_write_kb': 27640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3311}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:43.402 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:56:43.402 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'6879", 'reported_seq': 7062, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:42.660210+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:56:42.660210+0000', 'last_peered': '2024-09-18T11:56:42.660210+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:56:42.660210+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 6879, 'log_dups_size': 0, 'ondisk_log_size': 6879, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28160000, 'num_objects': 6875, 'num_object_clones': 0, 'num_object_copies': 13750, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3737, 'num_write_kb': 14948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 3339}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:43.402 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:56:43.402 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'6786", 'reported_seq': 6865, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:41.708090+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:56:41.708090+0000', 'last_peered': '2024-09-18T11:56:41.708090+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:56:41.708090+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6786, 'log_dups_size': 0, 'ondisk_log_size': 6786, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27776210, 'num_objects': 6783, 'num_object_clones': 0, 'num_object_copies': 13566, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6784, 'num_write_kb': 27127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3256}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:43.402 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:56:43.402 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'6869", 'reported_seq': 6953, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:41.687575+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:56:41.687575+0000', 'last_peered': '2024-09-18T11:56:41.687575+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:56:41.687575+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6869, 'log_dups_size': 0, 'ondisk_log_size': 6869, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28112082, 'num_objects': 6865, 'num_object_clones': 0, 'num_object_copies': 13730, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6867, 'num_write_kb': 27456, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3306}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:43.403 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:56:43.403 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'6842", 'reported_seq': 12574, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:42.666370+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:56:42.666370+0000', 'last_peered': '2024-09-18T11:56:42.666370+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:56:42.666370+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 6842, 'log_dups_size': 0, 'ondisk_log_size': 6842, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27997394, 'num_objects': 6837, 'num_object_clones': 0, 'num_object_copies': 13674, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 689, 'num_objects_unfound': 0, 'num_objects_dirty': 6837, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6840, 'num_write_kb': 27344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2793, 'num_bytes_recovered': 11423744, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 689}, {'shards': '2', 'objects': 6147}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:43.403 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:56:43.403 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'6876", 'reported_seq': 6952, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:41.731801+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:56:41.731801+0000', 'last_peered': '2024-09-18T11:56:41.731801+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:56:41.731801+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6876, 'log_dups_size': 0, 'ondisk_log_size': 6876, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28132562, 'num_objects': 6870, 'num_object_clones': 0, 'num_object_copies': 13740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6874, 'num_write_kb': 27476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3338}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:43.403 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:56:43.403 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'6910", 'reported_seq': 6964, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:42.643099+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:56:42.643099+0000', 'last_peered': '2024-09-18T11:56:42.643099+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:56:42.643099+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6910, 'log_dups_size': 0, 'ondisk_log_size': 6910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28284114, 'num_objects': 6907, 'num_object_clones': 0, 'num_object_copies': 13814, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6907, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6908, 'num_write_kb': 27623, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 3357}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:43.403 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:56:43.403 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'6859", 'reported_seq': 7066, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:42.612892+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:56:42.612892+0000', 'last_peered': '2024-09-18T11:56:42.612892+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:56:42.612892+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 6859, 'log_dups_size': 0, 'ondisk_log_size': 6859, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28069888, 'num_objects': 6853, 'num_object_clones': 0, 'num_object_copies': 13706, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6853, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3339, 'num_write_kb': 13356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 3260}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:43.403 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:56:43.403 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'6800", 'reported_seq': 6880, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:41.783159+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:56:41.783159+0000', 'last_peered': '2024-09-18T11:56:41.783159+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:56:41.783159+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6800, 'log_dups_size': 0, 'ondisk_log_size': 6800, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27845225, 'num_objects': 6799, 'num_object_clones': 0, 'num_object_copies': 13598, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6799, 'num_write_kb': 27193, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3264}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:43.404 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:56:43.404 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'6925", 'reported_seq': 7007, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:41.742067+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:56:41.742067+0000', 'last_peered': '2024-09-18T11:56:41.742067+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:56:41.742067+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6925, 'log_dups_size': 0, 'ondisk_log_size': 6925, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28333883, 'num_objects': 6920, 'num_object_clones': 0, 'num_object_copies': 13840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6920, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6922, 'num_write_kb': 27672, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3398}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:43.404 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:56:43.404 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'6793", 'reported_seq': 6872, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:41.752317+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:56:41.752317+0000', 'last_peered': '2024-09-18T11:56:41.752317+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:56:41.752317+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6793, 'log_dups_size': 0, 'ondisk_log_size': 6793, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27808978, 'num_objects': 6791, 'num_object_clones': 0, 'num_object_copies': 13582, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6791, 'num_write_kb': 27158, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3198}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:43.404 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:56:43.404 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'6908", 'reported_seq': 6990, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:41.771132+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:56:41.771132+0000', 'last_peered': '2024-09-18T11:56:41.771132+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:56:41.771132+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6908, 'log_dups_size': 0, 'ondisk_log_size': 6908, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28248484, 'num_objects': 6900, 'num_object_clones': 0, 'num_object_copies': 13800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6905, 'num_write_kb': 27591, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3377}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:43.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:44.134 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 138 20 110715 110695 3.13238 1.82031 0.0688383 0.0249316 2024-09-18T11:56:44.142 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:44.142 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/219564 objects degraded (17.803%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":204,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":109782,"data_bytes":449589319,"bytes_used":1842507776,"bytes_avail":288067784704,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":219564,"degraded_ratio":0.17803465048915121,"misplaced_objects":689,"misplaced_total":219564,"misplaced_ratio":0.0031380372009983419,"recovering_objects_per_sec":48,"recovering_bytes_per_sec":198608,"recovering_keys_per_sec":0,"num_objects_recovered":582,"num_bytes_recovered":2383872,"num_keys_recovered":0,"write_bytes_sec":2555298,"read_op_per_sec":0,"write_op_per_sec":623},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:44.166 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:44.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:44.943 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:44.973 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:44.994 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-18T11:56:43.073117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":449589319,"num_objects":109782,"num_object_clones":0,"num_object_copies":219564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":689,"num_objects_unfound":0,"num_objects_dirty":109782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93837,"num_write_kb":375222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2814,"num_bytes_recovered":11432572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109845,"ondisk_log_size":109845,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1799324,"kb_used_data":889156,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281316196,"statfs":{"total":289910292480,"available":288067784704,"internally_reserved":0,"allocated":910495744,"data_stored":909779452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,6,7,10,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30670848,"num_objects":7488,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-582,"num_objects_unfound":0,"num_objects_dirty":7488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":29952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":582,"num_bytes_recovered":2383872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002844"},"pg_stats":[{"pgid":"2.f","version":"37'6910","reported_seq":6979,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.709971+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:41.709971+0000","last_peered":"2024-09-18T11:56:41.709971+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:41.709971+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6910,"log_dups_size":0,"ondisk_log_size":6910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28303360,"num_objects":6910,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6910,"num_write_kb":27640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3311},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6895","reported_seq":7051,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:42.626646+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:42.626646+0000","last_peered":"2024-09-18T11:56:42.626646+0000","last_clean":"2024-09-18T11:56:42.626646+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:42.626646+0000","last_undegraded":"2024-09-18T11:56:42.626646+0000","last_fullsized":"2024-09-18T11:56:42.626646+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28217344,"num_objects":6889,"num_object_clones":0,"num_object_copies":13778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":16348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6879","reported_seq":7062,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:42.660210+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:42.660210+0000","last_peered":"2024-09-18T11:56:42.660210+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:42.660210+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6879,"log_dups_size":0,"ondisk_log_size":6879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28160000,"num_objects":6875,"num_object_clones":0,"num_object_copies":13750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":14948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3339}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6801","reported_seq":6828,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:42.524225+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:42.524225+0000","last_peered":"2024-09-18T11:56:42.524225+0000","last_clean":"2024-09-18T11:56:42.524225+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:42.524225+0000","last_undegraded":"2024-09-18T11:56:42.524225+0000","last_fullsized":"2024-09-18T11:56:42.524225+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6801,"log_dups_size":0,"ondisk_log_size":6801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27856896,"num_objects":6801,"num_object_clones":0,"num_object_copies":13602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6801,"num_write_kb":27204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'6866","reported_seq":7055,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:41.707867+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:41.707867+0000","last_peered":"2024-09-18T11:56:41.707867+0000","last_clean":"2024-09-18T11:56:41.707867+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:41.707867+0000","last_undegraded":"2024-09-18T11:56:41.707867+0000","last_fullsized":"2024-09-18T11:56:41.707867+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":6866,"log_dups_size":0,"ondisk_log_size":6866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28094464,"num_objects":6859,"num_object_clones":0,"num_object_copies":13718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4031,"num_write_kb":16124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6786","reported_seq":6865,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.708090+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:41.708090+0000","last_peered":"2024-09-18T11:56:41.708090+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:41.708090+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6786,"log_dups_size":0,"ondisk_log_size":6786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27776210,"num_objects":6783,"num_object_clones":0,"num_object_copies":13566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6784,"num_write_kb":27127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3256},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'6869","reported_seq":6953,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.687575+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:41.687575+0000","last_peered":"2024-09-18T11:56:41.687575+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:41.687575+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6869,"log_dups_size":0,"ondisk_log_size":6869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28112082,"num_objects":6865,"num_object_clones":0,"num_object_copies":13730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":27456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3306},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'6923","reported_seq":7146,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:41.766497+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:41.766497+0000","last_peered":"2024-09-18T11:56:41.766497+0000","last_clean":"2024-09-18T11:56:41.766497+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:41.766497+0000","last_undegraded":"2024-09-18T11:56:41.766497+0000","last_fullsized":"2024-09-18T11:56:41.766497+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":6923,"log_dups_size":0,"ondisk_log_size":6923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28348416,"num_objects":6921,"num_object_clones":0,"num_object_copies":13842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3240,"num_write_kb":12960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6842","reported_seq":12574,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:42.666370+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:42.666370+0000","last_peered":"2024-09-18T11:56:42.666370+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:42.666370+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6842,"log_dups_size":0,"ondisk_log_size":6842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27997394,"num_objects":6837,"num_object_clones":0,"num_object_copies":13674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":689,"num_objects_unfound":0,"num_objects_dirty":6837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6840,"num_write_kb":27344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2793,"num_bytes_recovered":11423744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":689},{"shards":"2","objects":6147}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'6876","reported_seq":6952,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.731801+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:41.731801+0000","last_peered":"2024-09-18T11:56:41.731801+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:41.731801+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6876,"log_dups_size":0,"ondisk_log_size":6876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28132562,"num_objects":6870,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6874,"num_write_kb":27476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3338},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6910","reported_seq":6964,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:42.643099+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:42.643099+0000","last_peered":"2024-09-18T11:56:42.643099+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:42.643099+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6910,"log_dups_size":0,"ondisk_log_size":6910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28284114,"num_objects":6907,"num_object_clones":0,"num_object_copies":13814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":27623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3357}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6859","reported_seq":7066,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:42.612892+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:42.612892+0000","last_peered":"2024-09-18T11:56:42.612892+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:42.612892+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6859,"log_dups_size":0,"ondisk_log_size":6859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28069888,"num_objects":6853,"num_object_clones":0,"num_object_copies":13706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3339,"num_write_kb":13356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6800","reported_seq":6880,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.783159+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:41.783159+0000","last_peered":"2024-09-18T11:56:41.783159+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:41.783159+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6800,"log_dups_size":0,"ondisk_log_size":6800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27845225,"num_objects":6799,"num_object_clones":0,"num_object_copies":13598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6799,"num_write_kb":27193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3264},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'6925","reported_seq":7007,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.742067+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:41.742067+0000","last_peered":"2024-09-18T11:56:41.742067+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:41.742067+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6925,"log_dups_size":0,"ondisk_log_size":6925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28333883,"num_objects":6920,"num_object_clones":0,"num_object_copies":13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6922,"num_write_kb":27672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3398},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6793","reported_seq":6872,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.752317+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:41.752317+0000","last_peered":"2024-09-18T11:56:41.752317+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:41.752317+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6793,"log_dups_size":0,"ondisk_log_size":6793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27808978,"num_objects":6791,"num_object_clones":0,"num_object_copies":13582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6791,"num_write_kb":27158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3198},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'6908","reported_seq":6990,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:41.771132+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:41.771132+0000","last_peered":"2024-09-18T11:56:41.771132+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:41.771132+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6908,"log_dups_size":0,"ondisk_log_size":6908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28248484,"num_objects":6900,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6905,"num_write_kb":27591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3377},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":449589300,"num_objects":109780,"num_object_clones":0,"num_object_copies":219560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":689,"num_objects_unfound":0,"num_objects_dirty":109780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93835,"num_write_kb":375220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2813,"num_bytes_recovered":11432553,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":910360576,"data_stored":910280559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109842,"ondisk_log_size":109842,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051618,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644032,"kb_used_data":333688,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93727808,"statfs":{"total":96636764160,"available":95977275392,"internally_reserved":0,"allocated":341696512,"data_stored":341483356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.318}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":1.018},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.42599999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855381,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468008,"kb_used_data":170144,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93903832,"statfs":{"total":96636764160,"available":96157523968,"internally_reserved":0,"allocated":174227456,"data_stored":173941098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.253}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":2.7080000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.27000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687284,"kb_used_data":385324,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93684556,"statfs":{"total":96636764160,"available":95932985344,"internally_reserved":0,"allocated":394571776,"data_stored":394354998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":4.4109999999999996},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":4.3319999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":4.46}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":394817536,"data_stored":394814057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173981696,"data_stored":173908637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341561344,"data_stored":341557865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:45.134 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 139 20 111198 111178 3.12342 1.88672 0.0578946 0.0250041 2024-09-18T11:56:46.134 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:2024-09-18T11:56:46.135012+0000 min lat: 0.0021114 max lat: 2.57663 avg lat: 0.0250643 2024-09-18T11:56:46.134 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:56:46.134 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 140 20 111714 111694 3.11551 2.01562 0.0165148 0.0250643 2024-09-18T11:56:47.136 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 141 20 112217 112197 3.10731 1.96484 0.0379103 0.0251338 2024-09-18T11:56:47.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:48.136 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 142 20 112699 112679 3.09869 1.88281 0.090297 0.0252022 2024-09-18T11:56:48.731 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:48.758 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:48.768 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-09-18T11:56:47.074095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":455733319,"num_objects":111282,"num_object_clones":0,"num_object_copies":222564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":689,"num_objects_unfound":0,"num_objects_dirty":111282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95337,"num_write_kb":381222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2814,"num_bytes_recovered":11432572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111345,"ondisk_log_size":111345,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1831228,"kb_used_data":904420,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281284292,"statfs":{"total":289910292480,"available":288035115008,"internally_reserved":0,"allocated":926126080,"data_stored":925409788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,6,6,10,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26509312,"num_objects":6472,"num_object_clones":0,"num_object_copies":12944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-399,"num_objects_unfound":0,"num_objects_dirty":6472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6472,"num_write_kb":25888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":399,"num_bytes_recovered":1634304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002862"},"pg_stats":[{"pgid":"2.f","version":"37'7080","reported_seq":7146,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.775604+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:46.775604+0000","last_peered":"2024-09-18T11:56:46.775604+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:46.775604+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7080,"log_dups_size":0,"ondisk_log_size":7080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28999680,"num_objects":7080,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7080,"num_write_kb":28320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3481},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'6895","reported_seq":7051,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:42.626646+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:42.626646+0000","last_peered":"2024-09-18T11:56:42.626646+0000","last_clean":"2024-09-18T11:56:42.626646+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:42.626646+0000","last_undegraded":"2024-09-18T11:56:42.626646+0000","last_fullsized":"2024-09-18T11:56:42.626646+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28217344,"num_objects":6889,"num_object_clones":0,"num_object_copies":13778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":16348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'6879","reported_seq":7062,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:42.660210+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:42.660210+0000","last_peered":"2024-09-18T11:56:42.660210+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:42.660210+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":6879,"log_dups_size":0,"ondisk_log_size":6879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28160000,"num_objects":6875,"num_object_clones":0,"num_object_copies":13750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3737,"num_write_kb":14948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3339}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6801","reported_seq":6828,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:42.524225+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:42.524225+0000","last_peered":"2024-09-18T11:56:42.524225+0000","last_clean":"2024-09-18T11:56:42.524225+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:42.524225+0000","last_undegraded":"2024-09-18T11:56:42.524225+0000","last_fullsized":"2024-09-18T11:56:42.524225+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6801,"log_dups_size":0,"ondisk_log_size":6801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27856896,"num_objects":6801,"num_object_clones":0,"num_object_copies":13602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6801,"num_write_kb":27204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'7010","reported_seq":7199,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:46.664217+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:46.664217+0000","last_peered":"2024-09-18T11:56:46.664217+0000","last_clean":"2024-09-18T11:56:46.664217+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:46.664217+0000","last_undegraded":"2024-09-18T11:56:46.664217+0000","last_fullsized":"2024-09-18T11:56:46.664217+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7010,"log_dups_size":0,"ondisk_log_size":7010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28684288,"num_objects":7003,"num_object_clones":0,"num_object_copies":14006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":16700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6934","reported_seq":7013,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.764647+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:46.764647+0000","last_peered":"2024-09-18T11:56:46.764647+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:46.764647+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6934,"log_dups_size":0,"ondisk_log_size":6934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28382418,"num_objects":6931,"num_object_clones":0,"num_object_copies":13862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6932,"num_write_kb":27719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3404},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'7020","reported_seq":7105,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.777370+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:46.777370+0000","last_peered":"2024-09-18T11:56:46.777370+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:46.777370+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7020,"log_dups_size":0,"ondisk_log_size":7020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28730578,"num_objects":7016,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7018,"num_write_kb":28060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3457},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'7079","reported_seq":7302,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:46.747916+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:46.747916+0000","last_peered":"2024-09-18T11:56:46.747916+0000","last_clean":"2024-09-18T11:56:46.747916+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:46.747916+0000","last_undegraded":"2024-09-18T11:56:46.747916+0000","last_fullsized":"2024-09-18T11:56:46.747916+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28987392,"num_objects":7077,"num_object_clones":0,"num_object_copies":14154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3396,"num_write_kb":13584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'6842","reported_seq":12574,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:42.666370+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:42.666370+0000","last_peered":"2024-09-18T11:56:42.666370+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:42.666370+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":6842,"log_dups_size":0,"ondisk_log_size":6842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27997394,"num_objects":6837,"num_object_clones":0,"num_object_copies":13674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":689,"num_objects_unfound":0,"num_objects_dirty":6837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6840,"num_write_kb":27344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2793,"num_bytes_recovered":11423744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":689},{"shards":"2","objects":6147}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'7039","reported_seq":7115,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.676692+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:46.676692+0000","last_peered":"2024-09-18T11:56:46.676692+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:46.676692+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28800210,"num_objects":7033,"num_object_clones":0,"num_object_copies":14066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7037,"num_write_kb":28128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3501},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'6910","reported_seq":6964,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:42.643099+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:42.643099+0000","last_peered":"2024-09-18T11:56:42.643099+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:42.643099+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6910,"log_dups_size":0,"ondisk_log_size":6910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28284114,"num_objects":6907,"num_object_clones":0,"num_object_copies":13814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":27623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3357}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'6859","reported_seq":7066,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:42.612892+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:42.612892+0000","last_peered":"2024-09-18T11:56:42.612892+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:42.612892+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":6859,"log_dups_size":0,"ondisk_log_size":6859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28069888,"num_objects":6853,"num_object_clones":0,"num_object_copies":13706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3339,"num_write_kb":13356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6945","reported_seq":7025,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.775568+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:46.775568+0000","last_peered":"2024-09-18T11:56:46.775568+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:46.775568+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6945,"log_dups_size":0,"ondisk_log_size":6945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28439145,"num_objects":6944,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6944,"num_write_kb":27773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3409},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'7073","reported_seq":7155,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.766391+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:46.766391+0000","last_peered":"2024-09-18T11:56:46.766391+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:46.766391+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7073,"log_dups_size":0,"ondisk_log_size":7073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28940091,"num_objects":7068,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7070,"num_write_kb":28264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3546},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6931","reported_seq":7009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.760934+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:46.760934+0000","last_peered":"2024-09-18T11:56:46.760934+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:46.760934+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6931,"log_dups_size":0,"ondisk_log_size":6931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28374226,"num_objects":6929,"num_object_clones":0,"num_object_copies":13858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6929,"num_write_kb":27710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3336},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'7045","reported_seq":7128,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.782509+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:46.782509+0000","last_peered":"2024-09-18T11:56:46.782509+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:46.782509+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28809636,"num_objects":7037,"num_object_clones":0,"num_object_copies":14074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7042,"num_write_kb":28139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3514},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":455733300,"num_objects":111280,"num_object_clones":0,"num_object_copies":222560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":689,"num_objects_unfound":0,"num_objects_dirty":111280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95335,"num_write_kb":381220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2813,"num_bytes_recovered":11432553,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":926134272,"data_stored":926054255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111342,"ondisk_log_size":111342,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051618,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644032,"kb_used_data":333688,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93727808,"statfs":{"total":96636764160,"available":95977275392,"internally_reserved":0,"allocated":341696512,"data_stored":341483356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.318}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":1.018},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.42599999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855382,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484244,"kb_used_data":178060,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93887596,"statfs":{"total":96636764160,"available":96140898304,"internally_reserved":0,"allocated":182333440,"data_stored":182047082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.253}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":2.7080000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.27000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702952,"kb_used_data":392672,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93668888,"statfs":{"total":96636764160,"available":95916941312,"internally_reserved":0,"allocated":402096128,"data_stored":401879350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.28199999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.246},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":402255872,"data_stored":402252393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182317056,"data_stored":182243997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341561344,"data_stored":341557865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:48.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:49.136 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 143 20 113151 113131 3.08937 1.76562 0.0134763 0.0252807 2024-09-18T11:56:49.312 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:49.317 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:49.341 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-18T11:56:49.074613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459542599,"num_objects":112212,"num_object_clones":0,"num_object_copies":224424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":478,"num_objects_unfound":0,"num_objects_dirty":112212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96267,"num_write_kb":384942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3025,"num_bytes_recovered":12296828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112275,"ondisk_log_size":112275,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1823680,"kb_used_data":909352,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281291840,"statfs":{"total":289910292480,"available":288042844160,"internally_reserved":0,"allocated":931176448,"data_stored":930460156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,4,6,9,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23846912,"num_objects":5822,"num_object_clones":0,"num_object_copies":11644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-610,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5822,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":610,"num_bytes_recovered":2498560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"2.f","version":"37'7080","reported_seq":7146,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.775604+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:46.775604+0000","last_peered":"2024-09-18T11:56:46.775604+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:46.775604+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7080,"log_dups_size":0,"ondisk_log_size":7080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28999680,"num_objects":7080,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7080,"num_write_kb":28320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3481},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'7041","reported_seq":7197,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:47.590839+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:47.590839+0000","last_peered":"2024-09-18T11:56:47.590839+0000","last_clean":"2024-09-18T11:56:47.590839+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:47.590839+0000","last_undegraded":"2024-09-18T11:56:47.590839+0000","last_fullsized":"2024-09-18T11:56:47.590839+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28815360,"num_objects":7035,"num_object_clones":0,"num_object_copies":14070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4233,"num_write_kb":16932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'7029","reported_seq":7212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:47.599723+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:47.599723+0000","last_peered":"2024-09-18T11:56:47.599723+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:47.599723+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7029,"log_dups_size":0,"ondisk_log_size":7029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28774400,"num_objects":7025,"num_object_clones":0,"num_object_copies":14050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3887,"num_write_kb":15548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3489}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6957","reported_seq":6984,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:47.686228+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:47.686228+0000","last_peered":"2024-09-18T11:56:47.686228+0000","last_clean":"2024-09-18T11:56:47.686228+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:47.686228+0000","last_undegraded":"2024-09-18T11:56:47.686228+0000","last_fullsized":"2024-09-18T11:56:47.686228+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6957,"log_dups_size":0,"ondisk_log_size":6957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28495872,"num_objects":6957,"num_object_clones":0,"num_object_copies":13914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6957,"num_write_kb":27828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'7010","reported_seq":7199,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:46.664217+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:46.664217+0000","last_peered":"2024-09-18T11:56:46.664217+0000","last_clean":"2024-09-18T11:56:46.664217+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:46.664217+0000","last_undegraded":"2024-09-18T11:56:46.664217+0000","last_fullsized":"2024-09-18T11:56:46.664217+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7010,"log_dups_size":0,"ondisk_log_size":7010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28684288,"num_objects":7003,"num_object_clones":0,"num_object_copies":14006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":16700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6934","reported_seq":7013,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.764647+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:46.764647+0000","last_peered":"2024-09-18T11:56:46.764647+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:46.764647+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6934,"log_dups_size":0,"ondisk_log_size":6934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28382418,"num_objects":6931,"num_object_clones":0,"num_object_copies":13862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6932,"num_write_kb":27719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3404},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'7020","reported_seq":7105,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.777370+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:46.777370+0000","last_peered":"2024-09-18T11:56:46.777370+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:46.777370+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7020,"log_dups_size":0,"ondisk_log_size":7020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28730578,"num_objects":7016,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7018,"num_write_kb":28060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3457},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'7079","reported_seq":7302,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:46.747916+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:46.747916+0000","last_peered":"2024-09-18T11:56:46.747916+0000","last_clean":"2024-09-18T11:56:46.747916+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:46.747916+0000","last_undegraded":"2024-09-18T11:56:46.747916+0000","last_fullsized":"2024-09-18T11:56:46.747916+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28987392,"num_objects":7077,"num_object_clones":0,"num_object_copies":14154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3396,"num_write_kb":13584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'7012","reported_seq":13168,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:47.684218+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:47.684218+0000","last_peered":"2024-09-18T11:56:47.684218+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:47.684218+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7012,"log_dups_size":0,"ondisk_log_size":7012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28693714,"num_objects":7007,"num_object_clones":0,"num_object_copies":14014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":478,"num_objects_unfound":0,"num_objects_dirty":7007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7010,"num_write_kb":28024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3004,"num_bytes_recovered":12288000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":478},{"shards":"2","objects":6528}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'7039","reported_seq":7115,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.676692+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:46.676692+0000","last_peered":"2024-09-18T11:56:46.676692+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:46.676692+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28800210,"num_objects":7033,"num_object_clones":0,"num_object_copies":14066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7037,"num_write_kb":28128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3501},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'7055","reported_seq":7109,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:47.621264+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:47.621264+0000","last_peered":"2024-09-18T11:56:47.621264+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:47.621264+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28878034,"num_objects":7052,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7053,"num_write_kb":28203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'7022","reported_seq":7229,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:47.637799+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:47.637799+0000","last_peered":"2024-09-18T11:56:47.637799+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:47.637799+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7022,"log_dups_size":0,"ondisk_log_size":7022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28737536,"num_objects":7016,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3502,"num_write_kb":14008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3423}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6945","reported_seq":7025,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.775568+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:46.775568+0000","last_peered":"2024-09-18T11:56:46.775568+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:46.775568+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6945,"log_dups_size":0,"ondisk_log_size":6945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28439145,"num_objects":6944,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6944,"num_write_kb":27773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3409},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'7073","reported_seq":7155,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.766391+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:46.766391+0000","last_peered":"2024-09-18T11:56:46.766391+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:46.766391+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7073,"log_dups_size":0,"ondisk_log_size":7073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28940091,"num_objects":7068,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7070,"num_write_kb":28264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3546},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6931","reported_seq":7009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.760934+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:46.760934+0000","last_peered":"2024-09-18T11:56:46.760934+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:46.760934+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6931,"log_dups_size":0,"ondisk_log_size":6931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28374226,"num_objects":6929,"num_object_clones":0,"num_object_copies":13858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6929,"num_write_kb":27710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3336},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'7045","reported_seq":7128,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.782509+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:46.782509+0000","last_peered":"2024-09-18T11:56:46.782509+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:46.782509+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28809636,"num_objects":7037,"num_object_clones":0,"num_object_copies":14074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7042,"num_write_kb":28139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3514},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":459542580,"num_objects":112210,"num_object_clones":0,"num_object_copies":224420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":478,"num_objects_unfound":0,"num_objects_dirty":112210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96265,"num_write_kb":384940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3024,"num_bytes_recovered":12296809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":931209216,"data_stored":931129199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112272,"ondisk_log_size":112272,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051619,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636484,"kb_used_data":338620,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93735356,"statfs":{"total":96636764160,"available":95985004544,"internally_reserved":0,"allocated":346746880,"data_stored":346533724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.27000000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":7.0190000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.374}]}]},{"osd":1,"up_from":35,"seq":150323855382,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484244,"kb_used_data":178060,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93887596,"statfs":{"total":96636764160,"available":96140898304,"internally_reserved":0,"allocated":182333440,"data_stored":182047082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.253}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":2.7080000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.27000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702952,"kb_used_data":392672,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93668888,"statfs":{"total":96636764160,"available":95916941312,"internally_reserved":0,"allocated":402096128,"data_stored":401879350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.28199999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.246},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":402255872,"data_stored":402252393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182317056,"data_stored":182243997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346636288,"data_stored":346632809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:49.343 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:56:49.343 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'7080", 'reported_seq': 7146, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:46.775604+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:56:46.775604+0000', 'last_peered': '2024-09-18T11:56:46.775604+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:56:46.775604+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7080, 'log_dups_size': 0, 'ondisk_log_size': 7080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28999680, 'num_objects': 7080, 'num_object_clones': 0, 'num_object_copies': 14160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7080, 'num_write_kb': 28320, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3481}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:49.343 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:56:49.343 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'7029", 'reported_seq': 7212, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:47.599723+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:56:47.599723+0000', 'last_peered': '2024-09-18T11:56:47.599723+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:56:47.599723+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 7029, 'log_dups_size': 0, 'ondisk_log_size': 7029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28774400, 'num_objects': 7025, 'num_object_clones': 0, 'num_object_copies': 14050, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3887, 'num_write_kb': 15548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 3489}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:49.343 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:56:49.343 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'6934", 'reported_seq': 7013, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:46.764647+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:56:46.764647+0000', 'last_peered': '2024-09-18T11:56:46.764647+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:56:46.764647+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6934, 'log_dups_size': 0, 'ondisk_log_size': 6934, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28382418, 'num_objects': 6931, 'num_object_clones': 0, 'num_object_copies': 13862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6932, 'num_write_kb': 27719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3404}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:49.343 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:56:49.343 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'7020", 'reported_seq': 7105, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:46.777370+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:56:46.777370+0000', 'last_peered': '2024-09-18T11:56:46.777370+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:56:46.777370+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7020, 'log_dups_size': 0, 'ondisk_log_size': 7020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28730578, 'num_objects': 7016, 'num_object_clones': 0, 'num_object_copies': 14032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7016, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7018, 'num_write_kb': 28060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3457}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:49.344 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:56:49.344 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'7012", 'reported_seq': 13168, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:47.684218+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:56:47.684218+0000', 'last_peered': '2024-09-18T11:56:47.684218+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:56:47.684218+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 7012, 'log_dups_size': 0, 'ondisk_log_size': 7012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28693714, 'num_objects': 7007, 'num_object_clones': 0, 'num_object_copies': 14014, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 478, 'num_objects_unfound': 0, 'num_objects_dirty': 7007, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7010, 'num_write_kb': 28024, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3004, 'num_bytes_recovered': 12288000, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 478}, {'shards': '2', 'objects': 6528}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:49.344 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:56:49.344 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'7039", 'reported_seq': 7115, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:46.676692+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:56:46.676692+0000', 'last_peered': '2024-09-18T11:56:46.676692+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:56:46.676692+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7039, 'log_dups_size': 0, 'ondisk_log_size': 7039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28800210, 'num_objects': 7033, 'num_object_clones': 0, 'num_object_copies': 14066, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7033, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7037, 'num_write_kb': 28128, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3501}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:49.344 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:56:49.344 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'7055", 'reported_seq': 7109, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:47.621264+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:56:47.621264+0000', 'last_peered': '2024-09-18T11:56:47.621264+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:56:47.621264+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7055, 'log_dups_size': 0, 'ondisk_log_size': 7055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28878034, 'num_objects': 7052, 'num_object_clones': 0, 'num_object_copies': 14104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7052, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7053, 'num_write_kb': 28203, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 3502}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:49.344 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:56:49.344 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'7022", 'reported_seq': 7229, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:47.637799+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:56:47.637799+0000', 'last_peered': '2024-09-18T11:56:47.637799+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:56:47.637799+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 7022, 'log_dups_size': 0, 'ondisk_log_size': 7022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28737536, 'num_objects': 7016, 'num_object_clones': 0, 'num_object_copies': 14032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7016, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3502, 'num_write_kb': 14008, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 3423}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:49.344 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:56:49.344 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'6945", 'reported_seq': 7025, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:46.775568+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:56:46.775568+0000', 'last_peered': '2024-09-18T11:56:46.775568+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:56:46.775568+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6945, 'log_dups_size': 0, 'ondisk_log_size': 6945, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28439145, 'num_objects': 6944, 'num_object_clones': 0, 'num_object_copies': 13888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6944, 'num_write_kb': 27773, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3409}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:49.345 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:56:49.345 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'7073", 'reported_seq': 7155, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:46.766391+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:56:46.766391+0000', 'last_peered': '2024-09-18T11:56:46.766391+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:56:46.766391+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7073, 'log_dups_size': 0, 'ondisk_log_size': 7073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28940091, 'num_objects': 7068, 'num_object_clones': 0, 'num_object_copies': 14136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7070, 'num_write_kb': 28264, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3546}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:49.345 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:56:49.345 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'6931", 'reported_seq': 7009, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:46.760934+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:56:46.760934+0000', 'last_peered': '2024-09-18T11:56:46.760934+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:56:46.760934+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 6931, 'log_dups_size': 0, 'ondisk_log_size': 6931, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28374226, 'num_objects': 6929, 'num_object_clones': 0, 'num_object_copies': 13858, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6929, 'num_write_kb': 27710, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3336}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:49.345 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:56:49.345 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'7045", 'reported_seq': 7128, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:46.782509+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:56:46.782509+0000', 'last_peered': '2024-09-18T11:56:46.782509+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:56:46.782509+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7045, 'log_dups_size': 0, 'ondisk_log_size': 7045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28809636, 'num_objects': 7037, 'num_object_clones': 0, 'num_object_copies': 14074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7042, 'num_write_kb': 28139, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3514}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:49.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:50.063 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:50.063 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/224424 objects degraded (17.418%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":210,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":112212,"data_bytes":459542599,"bytes_used":1867448320,"bytes_avail":288042844160,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":224424,"degraded_ratio":0.17417923216768261,"misplaced_objects":478,"misplaced_total":224424,"misplaced_ratio":0.0021298969807150752,"recovering_objects_per_sec":50,"recovering_bytes_per_sec":208163,"recovering_keys_per_sec":0,"num_objects_recovered":610,"num_bytes_recovered":2498560,"num_keys_recovered":0,"write_bytes_sec":1986764,"read_op_per_sec":0,"write_op_per_sec":485},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:50.105 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:50.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:50.136 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 144 20 113576 113556 3.07944 1.66016 0.0148893 0.0253601 2024-09-18T11:56:50.852 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:50.874 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:50.882 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-18T11:56:49.074613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459542599,"num_objects":112212,"num_object_clones":0,"num_object_copies":224424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":478,"num_objects_unfound":0,"num_objects_dirty":112212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96267,"num_write_kb":384942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3025,"num_bytes_recovered":12296828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112275,"ondisk_log_size":112275,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1823680,"kb_used_data":909352,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281291840,"statfs":{"total":289910292480,"available":288042844160,"internally_reserved":0,"allocated":931176448,"data_stored":930460156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,4,6,9,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23846912,"num_objects":5822,"num_object_clones":0,"num_object_copies":11644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-610,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5822,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":610,"num_bytes_recovered":2498560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"2.f","version":"37'7080","reported_seq":7146,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.775604+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:46.775604+0000","last_peered":"2024-09-18T11:56:46.775604+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:46.775604+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7080,"log_dups_size":0,"ondisk_log_size":7080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28999680,"num_objects":7080,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7080,"num_write_kb":28320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3481},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'7041","reported_seq":7197,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:47.590839+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:47.590839+0000","last_peered":"2024-09-18T11:56:47.590839+0000","last_clean":"2024-09-18T11:56:47.590839+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:47.590839+0000","last_undegraded":"2024-09-18T11:56:47.590839+0000","last_fullsized":"2024-09-18T11:56:47.590839+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28815360,"num_objects":7035,"num_object_clones":0,"num_object_copies":14070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4233,"num_write_kb":16932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'7029","reported_seq":7212,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:47.599723+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:47.599723+0000","last_peered":"2024-09-18T11:56:47.599723+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:47.599723+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7029,"log_dups_size":0,"ondisk_log_size":7029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28774400,"num_objects":7025,"num_object_clones":0,"num_object_copies":14050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3887,"num_write_kb":15548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3489}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'6957","reported_seq":6984,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:47.686228+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:47.686228+0000","last_peered":"2024-09-18T11:56:47.686228+0000","last_clean":"2024-09-18T11:56:47.686228+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:47.686228+0000","last_undegraded":"2024-09-18T11:56:47.686228+0000","last_fullsized":"2024-09-18T11:56:47.686228+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6957,"log_dups_size":0,"ondisk_log_size":6957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28495872,"num_objects":6957,"num_object_clones":0,"num_object_copies":13914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6957,"num_write_kb":27828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'7010","reported_seq":7199,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:46.664217+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:46.664217+0000","last_peered":"2024-09-18T11:56:46.664217+0000","last_clean":"2024-09-18T11:56:46.664217+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:46.664217+0000","last_undegraded":"2024-09-18T11:56:46.664217+0000","last_fullsized":"2024-09-18T11:56:46.664217+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7010,"log_dups_size":0,"ondisk_log_size":7010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28684288,"num_objects":7003,"num_object_clones":0,"num_object_copies":14006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":16700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'6934","reported_seq":7013,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.764647+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:46.764647+0000","last_peered":"2024-09-18T11:56:46.764647+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:46.764647+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6934,"log_dups_size":0,"ondisk_log_size":6934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28382418,"num_objects":6931,"num_object_clones":0,"num_object_copies":13862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6932,"num_write_kb":27719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3404},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'7020","reported_seq":7105,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.777370+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:46.777370+0000","last_peered":"2024-09-18T11:56:46.777370+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:46.777370+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7020,"log_dups_size":0,"ondisk_log_size":7020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28730578,"num_objects":7016,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7018,"num_write_kb":28060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3457},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'7079","reported_seq":7302,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:46.747916+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:46.747916+0000","last_peered":"2024-09-18T11:56:46.747916+0000","last_clean":"2024-09-18T11:56:46.747916+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:46.747916+0000","last_undegraded":"2024-09-18T11:56:46.747916+0000","last_fullsized":"2024-09-18T11:56:46.747916+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7079,"log_dups_size":0,"ondisk_log_size":7079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28987392,"num_objects":7077,"num_object_clones":0,"num_object_copies":14154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3396,"num_write_kb":13584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'7012","reported_seq":13168,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:47.684218+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:47.684218+0000","last_peered":"2024-09-18T11:56:47.684218+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:47.684218+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7012,"log_dups_size":0,"ondisk_log_size":7012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28693714,"num_objects":7007,"num_object_clones":0,"num_object_copies":14014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":478,"num_objects_unfound":0,"num_objects_dirty":7007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7010,"num_write_kb":28024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3004,"num_bytes_recovered":12288000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":478},{"shards":"2","objects":6528}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'7039","reported_seq":7115,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.676692+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:46.676692+0000","last_peered":"2024-09-18T11:56:46.676692+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:46.676692+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28800210,"num_objects":7033,"num_object_clones":0,"num_object_copies":14066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7037,"num_write_kb":28128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3501},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'7055","reported_seq":7109,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:47.621264+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:47.621264+0000","last_peered":"2024-09-18T11:56:47.621264+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:47.621264+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28878034,"num_objects":7052,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7053,"num_write_kb":28203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'7022","reported_seq":7229,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:47.637799+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:47.637799+0000","last_peered":"2024-09-18T11:56:47.637799+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:47.637799+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7022,"log_dups_size":0,"ondisk_log_size":7022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28737536,"num_objects":7016,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3502,"num_write_kb":14008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3423}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'6945","reported_seq":7025,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.775568+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:46.775568+0000","last_peered":"2024-09-18T11:56:46.775568+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:46.775568+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6945,"log_dups_size":0,"ondisk_log_size":6945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28439145,"num_objects":6944,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6944,"num_write_kb":27773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3409},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'7073","reported_seq":7155,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.766391+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:46.766391+0000","last_peered":"2024-09-18T11:56:46.766391+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:46.766391+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7073,"log_dups_size":0,"ondisk_log_size":7073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28940091,"num_objects":7068,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7070,"num_write_kb":28264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3546},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'6931","reported_seq":7009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.760934+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:46.760934+0000","last_peered":"2024-09-18T11:56:46.760934+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:46.760934+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":6931,"log_dups_size":0,"ondisk_log_size":6931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28374226,"num_objects":6929,"num_object_clones":0,"num_object_copies":13858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6929,"num_write_kb":27710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3336},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'7045","reported_seq":7128,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:46.782509+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:46.782509+0000","last_peered":"2024-09-18T11:56:46.782509+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:46.782509+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28809636,"num_objects":7037,"num_object_clones":0,"num_object_copies":14074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7042,"num_write_kb":28139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3514},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":459542580,"num_objects":112210,"num_object_clones":0,"num_object_copies":224420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":478,"num_objects_unfound":0,"num_objects_dirty":112210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96265,"num_write_kb":384940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3024,"num_bytes_recovered":12296809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":931209216,"data_stored":931129199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112272,"ondisk_log_size":112272,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051619,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636484,"kb_used_data":338620,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93735356,"statfs":{"total":96636764160,"available":95985004544,"internally_reserved":0,"allocated":346746880,"data_stored":346533724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.27000000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":7.0190000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.374}]}]},{"osd":1,"up_from":35,"seq":150323855382,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484244,"kb_used_data":178060,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93887596,"statfs":{"total":96636764160,"available":96140898304,"internally_reserved":0,"allocated":182333440,"data_stored":182047082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.253}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":2.7080000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.27000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702952,"kb_used_data":392672,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93668888,"statfs":{"total":96636764160,"available":95916941312,"internally_reserved":0,"allocated":402096128,"data_stored":401879350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.28199999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.246},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":402255872,"data_stored":402252393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182317056,"data_stored":182243997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346636288,"data_stored":346632809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:51.136 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 145 20 114046 114026 3.07087 1.83594 0.0840806 0.0254304 2024-09-18T11:56:52.136 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 146 20 114535 114515 3.06292 1.91016 0.0170158 0.0254989 2024-09-18T11:56:53.136 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 147 20 115020 115000 3.05497 1.89453 0.0154613 0.0255619 2024-09-18T11:56:53.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:54.136 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 148 20 115513 115493 3.04734 1.92578 0.0494773 0.0256298 2024-09-18T11:56:54.599 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:54.615 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:54.637 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-18T11:56:53.075492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":469041223,"num_objects":114531,"num_object_clones":0,"num_object_copies":229062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":193,"num_objects_unfound":0,"num_objects_dirty":114531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98586,"num_write_kb":394218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3310,"num_bytes_recovered":13464188,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114594,"ondisk_log_size":114594,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1839004,"kb_used_data":928836,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281276516,"statfs":{"total":289910292480,"available":288027152384,"internally_reserved":0,"allocated":951128064,"data_stored":950411772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,9,14,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29708288,"num_objects":7253,"num_object_clones":0,"num_object_copies":14506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-700,"num_objects_unfound":0,"num_objects_dirty":7253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7253,"num_write_kb":29012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":700,"num_bytes_recovered":2867200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002817"},"pg_stats":[{"pgid":"2.f","version":"37'7213","reported_seq":7281,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.817639+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:51.817639+0000","last_peered":"2024-09-18T11:56:51.817639+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:51.817639+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29544448,"num_objects":7213,"num_object_clones":0,"num_object_copies":14426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7213,"num_write_kb":28852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3614},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'7168","reported_seq":7324,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:52.605047+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:52.605047+0000","last_peered":"2024-09-18T11:56:52.605047+0000","last_clean":"2024-09-18T11:56:52.605047+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:52.605047+0000","last_undegraded":"2024-09-18T11:56:52.605047+0000","last_fullsized":"2024-09-18T11:56:52.605047+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7168,"log_dups_size":0,"ondisk_log_size":7168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29335552,"num_objects":7162,"num_object_clones":0,"num_object_copies":14324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":17440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'7183","reported_seq":7366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:52.654333+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:52.654333+0000","last_peered":"2024-09-18T11:56:52.654333+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:52.654333+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29405184,"num_objects":7179,"num_object_clones":0,"num_object_copies":14358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4041,"num_write_kb":16164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3643}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'7096","reported_seq":7123,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:52.635177+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:52.635177+0000","last_peered":"2024-09-18T11:56:52.635177+0000","last_clean":"2024-09-18T11:56:52.635177+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:52.635177+0000","last_undegraded":"2024-09-18T11:56:52.635177+0000","last_fullsized":"2024-09-18T11:56:52.635177+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7096,"log_dups_size":0,"ondisk_log_size":7096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29065216,"num_objects":7096,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7096,"num_write_kb":28384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'7148","reported_seq":7337,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:51.782787+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:51.782787+0000","last_peered":"2024-09-18T11:56:51.782787+0000","last_clean":"2024-09-18T11:56:51.782787+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:51.782787+0000","last_undegraded":"2024-09-18T11:56:51.782787+0000","last_fullsized":"2024-09-18T11:56:51.782787+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29249536,"num_objects":7141,"num_object_clones":0,"num_object_copies":14282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4313,"num_write_kb":17252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'7089","reported_seq":7168,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.812578+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:51.812578+0000","last_peered":"2024-09-18T11:56:51.812578+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:51.812578+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7089,"log_dups_size":0,"ondisk_log_size":7089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29017298,"num_objects":7086,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":28339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3559},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'7157","reported_seq":7242,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.698958+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:51.698958+0000","last_peered":"2024-09-18T11:56:51.698958+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:51.698958+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7157,"log_dups_size":0,"ondisk_log_size":7157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29291730,"num_objects":7153,"num_object_clones":0,"num_object_copies":14306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7155,"num_write_kb":28608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3594},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'7220","reported_seq":7443,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:51.787908+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:51.787908+0000","last_peered":"2024-09-18T11:56:51.787908+0000","last_clean":"2024-09-18T11:56:51.787908+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:51.787908+0000","last_undegraded":"2024-09-18T11:56:51.787908+0000","last_fullsized":"2024-09-18T11:56:51.787908+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7220,"log_dups_size":0,"ondisk_log_size":7220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29564928,"num_objects":7218,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'7151","reported_seq":13876,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:52.687181+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:52.687181+0000","last_peered":"2024-09-18T11:56:52.687181+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:52.687181+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7151,"log_dups_size":0,"ondisk_log_size":7151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29263058,"num_objects":7146,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":193,"num_objects_unfound":0,"num_objects_dirty":7146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7149,"num_write_kb":28580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3289,"num_bytes_recovered":13455360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":193},{"shards":"2","objects":6952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'7187","reported_seq":7263,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.814554+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:51.814554+0000","last_peered":"2024-09-18T11:56:51.814554+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:51.814554+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7187,"log_dups_size":0,"ondisk_log_size":7187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29406418,"num_objects":7181,"num_object_clones":0,"num_object_copies":14362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7185,"num_write_kb":28720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3649},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'7213","reported_seq":7267,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:52.667169+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:52.667169+0000","last_peered":"2024-09-18T11:56:52.667169+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:52.667169+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29525202,"num_objects":7210,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7211,"num_write_kb":28835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3660}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'7154","reported_seq":7361,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:52.635512+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:52.635512+0000","last_peered":"2024-09-18T11:56:52.635512+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:52.635512+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7154,"log_dups_size":0,"ondisk_log_size":7154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29278208,"num_objects":7148,"num_object_clones":0,"num_object_copies":14296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3634,"num_write_kb":14536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'7106","reported_seq":7185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.755944+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:51.755944+0000","last_peered":"2024-09-18T11:56:51.755944+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:51.755944+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7106,"log_dups_size":0,"ondisk_log_size":7106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29098601,"num_objects":7105,"num_object_clones":0,"num_object_copies":14210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":28417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3570},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'7225","reported_seq":7307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.733830+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:51.733830+0000","last_peered":"2024-09-18T11:56:51.733830+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:51.733830+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29562683,"num_objects":7220,"num_object_clones":0,"num_object_copies":14440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":28872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3698},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'7089","reported_seq":7168,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.795712+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:51.795712+0000","last_peered":"2024-09-18T11:56:51.795712+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:51.795712+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7089,"log_dups_size":0,"ondisk_log_size":7089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29021394,"num_objects":7087,"num_object_clones":0,"num_object_copies":14174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":28342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3494},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'7192","reported_seq":7274,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.722294+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:51.722294+0000","last_peered":"2024-09-18T11:56:51.722294+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:51.722294+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29411748,"num_objects":7184,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":28727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3661},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":469041204,"num_objects":114529,"num_object_clones":0,"num_object_copies":229058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":193,"num_objects_unfound":0,"num_objects_dirty":114529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98584,"num_write_kb":394216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3309,"num_bytes_recovered":13464169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":951455744,"data_stored":951375727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114591,"ondisk_log_size":114591,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051620,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645076,"kb_used_data":343052,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93726764,"statfs":{"total":96636764160,"available":95976206336,"internally_reserved":0,"allocated":351285248,"data_stored":351072092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,6,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":3.02}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":4.1970000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.42899999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855383,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496672,"kb_used_data":186328,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93875168,"statfs":{"total":96636764160,"available":96128172032,"internally_reserved":0,"allocated":190799872,"data_stored":190513514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.217},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.54000000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.313},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.78500000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697256,"kb_used_data":399456,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93674584,"statfs":{"total":96636764160,"available":95922774016,"internally_reserved":0,"allocated":409042944,"data_stored":408826166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":1.121}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409407488,"data_stored":409404009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":190787584,"data_stored":190714525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351260672,"data_stored":351257193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:54.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:55.138 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 149 20 115982 115962 3.03916 1.83203 0.027239 0.0256953 2024-09-18T11:56:55.383 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:55.406 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:55.413 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-18T11:56:55.079095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":469041223,"num_objects":114531,"num_object_clones":0,"num_object_copies":229062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":193,"num_objects_unfound":0,"num_objects_dirty":114531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98586,"num_write_kb":394218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3310,"num_bytes_recovered":13464188,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114594,"ondisk_log_size":114594,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1839004,"kb_used_data":928836,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281276516,"statfs":{"total":289910292480,"available":288027152384,"internally_reserved":0,"allocated":951128064,"data_stored":950411772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,9,14,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19451904,"num_objects":4749,"num_object_clones":0,"num_object_copies":9498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-496,"num_objects_unfound":0,"num_objects_dirty":4749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":18996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":496,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005977"},"pg_stats":[{"pgid":"2.f","version":"37'7213","reported_seq":7281,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.817639+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:51.817639+0000","last_peered":"2024-09-18T11:56:51.817639+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:51.817639+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29544448,"num_objects":7213,"num_object_clones":0,"num_object_copies":14426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7213,"num_write_kb":28852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3614},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'7168","reported_seq":7324,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:52.605047+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:52.605047+0000","last_peered":"2024-09-18T11:56:52.605047+0000","last_clean":"2024-09-18T11:56:52.605047+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:52.605047+0000","last_undegraded":"2024-09-18T11:56:52.605047+0000","last_fullsized":"2024-09-18T11:56:52.605047+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7168,"log_dups_size":0,"ondisk_log_size":7168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29335552,"num_objects":7162,"num_object_clones":0,"num_object_copies":14324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":17440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'7183","reported_seq":7366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:52.654333+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:52.654333+0000","last_peered":"2024-09-18T11:56:52.654333+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:52.654333+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29405184,"num_objects":7179,"num_object_clones":0,"num_object_copies":14358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4041,"num_write_kb":16164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3643}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'7096","reported_seq":7123,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:52.635177+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:52.635177+0000","last_peered":"2024-09-18T11:56:52.635177+0000","last_clean":"2024-09-18T11:56:52.635177+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:52.635177+0000","last_undegraded":"2024-09-18T11:56:52.635177+0000","last_fullsized":"2024-09-18T11:56:52.635177+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7096,"log_dups_size":0,"ondisk_log_size":7096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29065216,"num_objects":7096,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7096,"num_write_kb":28384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'7148","reported_seq":7337,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:51.782787+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:51.782787+0000","last_peered":"2024-09-18T11:56:51.782787+0000","last_clean":"2024-09-18T11:56:51.782787+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:51.782787+0000","last_undegraded":"2024-09-18T11:56:51.782787+0000","last_fullsized":"2024-09-18T11:56:51.782787+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29249536,"num_objects":7141,"num_object_clones":0,"num_object_copies":14282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4313,"num_write_kb":17252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'7089","reported_seq":7168,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.812578+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:51.812578+0000","last_peered":"2024-09-18T11:56:51.812578+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:51.812578+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7089,"log_dups_size":0,"ondisk_log_size":7089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29017298,"num_objects":7086,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":28339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3559},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'7157","reported_seq":7242,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.698958+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:51.698958+0000","last_peered":"2024-09-18T11:56:51.698958+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:51.698958+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7157,"log_dups_size":0,"ondisk_log_size":7157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29291730,"num_objects":7153,"num_object_clones":0,"num_object_copies":14306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7155,"num_write_kb":28608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3594},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'7220","reported_seq":7443,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:51.787908+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:51.787908+0000","last_peered":"2024-09-18T11:56:51.787908+0000","last_clean":"2024-09-18T11:56:51.787908+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:51.787908+0000","last_undegraded":"2024-09-18T11:56:51.787908+0000","last_fullsized":"2024-09-18T11:56:51.787908+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7220,"log_dups_size":0,"ondisk_log_size":7220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29564928,"num_objects":7218,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'7151","reported_seq":13876,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:52.687181+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:52.687181+0000","last_peered":"2024-09-18T11:56:52.687181+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:52.687181+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7151,"log_dups_size":0,"ondisk_log_size":7151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29263058,"num_objects":7146,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":193,"num_objects_unfound":0,"num_objects_dirty":7146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7149,"num_write_kb":28580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3289,"num_bytes_recovered":13455360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":193},{"shards":"2","objects":6952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'7187","reported_seq":7263,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.814554+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:51.814554+0000","last_peered":"2024-09-18T11:56:51.814554+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:51.814554+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7187,"log_dups_size":0,"ondisk_log_size":7187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29406418,"num_objects":7181,"num_object_clones":0,"num_object_copies":14362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7185,"num_write_kb":28720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3649},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'7213","reported_seq":7267,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:52.667169+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:52.667169+0000","last_peered":"2024-09-18T11:56:52.667169+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:52.667169+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29525202,"num_objects":7210,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7211,"num_write_kb":28835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3660}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'7154","reported_seq":7361,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:52.635512+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:52.635512+0000","last_peered":"2024-09-18T11:56:52.635512+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:52.635512+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7154,"log_dups_size":0,"ondisk_log_size":7154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29278208,"num_objects":7148,"num_object_clones":0,"num_object_copies":14296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3634,"num_write_kb":14536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'7106","reported_seq":7185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.755944+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:51.755944+0000","last_peered":"2024-09-18T11:56:51.755944+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:51.755944+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7106,"log_dups_size":0,"ondisk_log_size":7106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29098601,"num_objects":7105,"num_object_clones":0,"num_object_copies":14210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":28417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3570},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'7225","reported_seq":7307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.733830+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:51.733830+0000","last_peered":"2024-09-18T11:56:51.733830+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:51.733830+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29562683,"num_objects":7220,"num_object_clones":0,"num_object_copies":14440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":28872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3698},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'7089","reported_seq":7168,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.795712+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:51.795712+0000","last_peered":"2024-09-18T11:56:51.795712+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:51.795712+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7089,"log_dups_size":0,"ondisk_log_size":7089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29021394,"num_objects":7087,"num_object_clones":0,"num_object_copies":14174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":28342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3494},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'7192","reported_seq":7274,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.722294+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:51.722294+0000","last_peered":"2024-09-18T11:56:51.722294+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:51.722294+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29411748,"num_objects":7184,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":28727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3661},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":469041204,"num_objects":114529,"num_object_clones":0,"num_object_copies":229058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":193,"num_objects_unfound":0,"num_objects_dirty":114529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98584,"num_write_kb":394216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3309,"num_bytes_recovered":13464169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":951455744,"data_stored":951375727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114591,"ondisk_log_size":114591,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051620,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645076,"kb_used_data":343052,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93726764,"statfs":{"total":96636764160,"available":95976206336,"internally_reserved":0,"allocated":351285248,"data_stored":351072092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,6,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":3.02}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":4.1970000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.42899999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855383,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496672,"kb_used_data":186328,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93875168,"statfs":{"total":96636764160,"available":96128172032,"internally_reserved":0,"allocated":190799872,"data_stored":190513514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.217},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.54000000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.313},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.78500000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697256,"kb_used_data":399456,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93674584,"statfs":{"total":96636764160,"available":95922774016,"internally_reserved":0,"allocated":409042944,"data_stored":408826166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":1.121}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409407488,"data_stored":409404009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":190787584,"data_stored":190714525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351260672,"data_stored":351257193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:55.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:56:55.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "37'7213", 'reported_seq': 7281, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:51.817639+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:56:51.817639+0000', 'last_peered': '2024-09-18T11:56:51.817639+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:56:51.817639+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7213, 'log_dups_size': 0, 'ondisk_log_size': 7213, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29544448, 'num_objects': 7213, 'num_object_clones': 0, 'num_object_copies': 14426, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7213, 'num_write_kb': 28852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3614}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:55.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:56:55.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "37'7183", 'reported_seq': 7366, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:52.654333+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:56:52.654333+0000', 'last_peered': '2024-09-18T11:56:52.654333+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:56:52.654333+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 7183, 'log_dups_size': 0, 'ondisk_log_size': 7183, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29405184, 'num_objects': 7179, 'num_object_clones': 0, 'num_object_copies': 14358, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4041, 'num_write_kb': 16164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 3643}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:55.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:56:55.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "37'7089", 'reported_seq': 7168, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:51.812578+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:56:51.812578+0000', 'last_peered': '2024-09-18T11:56:51.812578+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:56:51.812578+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7089, 'log_dups_size': 0, 'ondisk_log_size': 7089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29017298, 'num_objects': 7086, 'num_object_clones': 0, 'num_object_copies': 14172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7087, 'num_write_kb': 28339, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3559}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:55.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:56:55.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "37'7157", 'reported_seq': 7242, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:51.698958+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:56:51.698958+0000', 'last_peered': '2024-09-18T11:56:51.698958+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:56:51.698958+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7157, 'log_dups_size': 0, 'ondisk_log_size': 7157, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29291730, 'num_objects': 7153, 'num_object_clones': 0, 'num_object_copies': 14306, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7155, 'num_write_kb': 28608, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3594}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:55.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:56:55.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'7151", 'reported_seq': 13876, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:52.687181+0000', 'last_change': '2024-09-18T11:55:04.628680+0000', 'last_active': '2024-09-18T11:56:52.687181+0000', 'last_peered': '2024-09-18T11:56:52.687181+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:56:52.687181+0000', 'last_undegraded': '2024-09-18T11:55:04.286738+0000', 'last_fullsized': '2024-09-18T11:55:04.246740+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 7151, 'log_dups_size': 0, 'ondisk_log_size': 7151, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29263058, 'num_objects': 7146, 'num_object_clones': 0, 'num_object_copies': 14292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 193, 'num_objects_unfound': 0, 'num_objects_dirty': 7146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7149, 'num_write_kb': 28580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3289, 'num_bytes_recovered': 13455360, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 193}, {'shards': '2', 'objects': 6952}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:55.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:56:55.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "37'7187", 'reported_seq': 7263, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:51.814554+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:56:51.814554+0000', 'last_peered': '2024-09-18T11:56:51.814554+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:56:51.814554+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7187, 'log_dups_size': 0, 'ondisk_log_size': 7187, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29406418, 'num_objects': 7181, 'num_object_clones': 0, 'num_object_copies': 14362, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7185, 'num_write_kb': 28720, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3649}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:55.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:56:55.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "37'7213", 'reported_seq': 7267, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:52.667169+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:56:52.667169+0000', 'last_peered': '2024-09-18T11:56:52.667169+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:56:52.667169+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7213, 'log_dups_size': 0, 'ondisk_log_size': 7213, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29525202, 'num_objects': 7210, 'num_object_clones': 0, 'num_object_copies': 14420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7211, 'num_write_kb': 28835, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 3660}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:55.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:56:55.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "37'7154", 'reported_seq': 7361, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:52.635512+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:56:52.635512+0000', 'last_peered': '2024-09-18T11:56:52.635512+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:56:52.635512+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 7154, 'log_dups_size': 0, 'ondisk_log_size': 7154, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29278208, 'num_objects': 7148, 'num_object_clones': 0, 'num_object_copies': 14296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7148, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3634, 'num_write_kb': 14536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 3555}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:56:55.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:56:55.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "37'7106", 'reported_seq': 7185, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:51.755944+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:56:51.755944+0000', 'last_peered': '2024-09-18T11:56:51.755944+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:56:51.755944+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7106, 'log_dups_size': 0, 'ondisk_log_size': 7106, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29098601, 'num_objects': 7105, 'num_object_clones': 0, 'num_object_copies': 14210, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7105, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7105, 'num_write_kb': 28417, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3570}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:55.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:56:55.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "37'7225", 'reported_seq': 7307, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:51.733830+0000', 'last_change': '2024-09-18T11:55:04.689591+0000', 'last_active': '2024-09-18T11:56:51.733830+0000', 'last_peered': '2024-09-18T11:56:51.733830+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:56:51.733830+0000', 'last_undegraded': '2024-09-18T11:55:04.322591+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7225, 'log_dups_size': 0, 'ondisk_log_size': 7225, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29562683, 'num_objects': 7220, 'num_object_clones': 0, 'num_object_copies': 14440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3525, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7222, 'num_write_kb': 28872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3698}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:55.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:56:55.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "37'7089", 'reported_seq': 7168, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:51.795712+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:56:51.795712+0000', 'last_peered': '2024-09-18T11:56:51.795712+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:56:51.795712+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7089, 'log_dups_size': 0, 'ondisk_log_size': 7089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29021394, 'num_objects': 7087, 'num_object_clones': 0, 'num_object_copies': 14174, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7087, 'num_write_kb': 28342, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3494}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:55.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:56:55.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "37'7192", 'reported_seq': 7274, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:51.722294+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:56:51.722294+0000', 'last_peered': '2024-09-18T11:56:51.722294+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:56:51.722294+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7192, 'log_dups_size': 0, 'ondisk_log_size': 7192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29411748, 'num_objects': 7184, 'num_object_clones': 0, 'num_object_copies': 14368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7189, 'num_write_kb': 28727, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3661}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:56:55.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:56:56.138 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 150 20 116454 116434 3.03119 1.84375 0.0350794 0.0257642 2024-09-18T11:56:56.361 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:56.361 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39090/229062 objects degraded (17.065%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":216,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":114531,"data_bytes":469041223,"bytes_used":1883140096,"bytes_avail":288027152384,"bytes_total":289910292480,"degraded_objects":39090,"degraded_total":229062,"degraded_ratio":0.17065248709956257,"misplaced_objects":193,"misplaced_total":229062,"misplaced_ratio":0.00084256664134601113,"recovering_objects_per_sec":41,"recovering_bytes_per_sec":169217,"recovering_keys_per_sec":0,"num_objects_recovered":496,"num_bytes_recovered":2031616,"num_keys_recovered":0,"write_bytes_sec":1620184,"read_op_per_sec":0,"write_op_per_sec":395},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:56:56.418 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:56:56.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:56:56.978 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:56:56.990 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:56:57.001 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-18T11:56:56.464803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":469041223,"num_objects":114531,"num_object_clones":0,"num_object_copies":229062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":193,"num_objects_unfound":0,"num_objects_dirty":114531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98586,"num_write_kb":394218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3310,"num_bytes_recovered":13464188,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114594,"ondisk_log_size":114594,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1839004,"kb_used_data":928836,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281276516,"statfs":{"total":289910292480,"available":288027152384,"internally_reserved":0,"allocated":951128064,"data_stored":950411772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,9,14,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19451904,"num_objects":4749,"num_object_clones":0,"num_object_copies":9498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-496,"num_objects_unfound":0,"num_objects_dirty":4749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":18996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":496,"num_bytes_recovered":2031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.391156"},"pg_stats":[{"pgid":"2.f","version":"37'7213","reported_seq":7281,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.817639+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:51.817639+0000","last_peered":"2024-09-18T11:56:51.817639+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:51.817639+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29544448,"num_objects":7213,"num_object_clones":0,"num_object_copies":14426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7213,"num_write_kb":28852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3614},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"37'7168","reported_seq":7324,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:52.605047+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:52.605047+0000","last_peered":"2024-09-18T11:56:52.605047+0000","last_clean":"2024-09-18T11:56:52.605047+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:52.605047+0000","last_undegraded":"2024-09-18T11:56:52.605047+0000","last_fullsized":"2024-09-18T11:56:52.605047+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7168,"log_dups_size":0,"ondisk_log_size":7168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29335552,"num_objects":7162,"num_object_clones":0,"num_object_copies":14324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":17440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"37'7183","reported_seq":7366,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:52.654333+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:52.654333+0000","last_peered":"2024-09-18T11:56:52.654333+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:52.654333+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29405184,"num_objects":7179,"num_object_clones":0,"num_object_copies":14358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4041,"num_write_kb":16164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3643}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"37'7096","reported_seq":7123,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:52.635177+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:52.635177+0000","last_peered":"2024-09-18T11:56:52.635177+0000","last_clean":"2024-09-18T11:56:52.635177+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:52.635177+0000","last_undegraded":"2024-09-18T11:56:52.635177+0000","last_fullsized":"2024-09-18T11:56:52.635177+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7096,"log_dups_size":0,"ondisk_log_size":7096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29065216,"num_objects":7096,"num_object_clones":0,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7096,"num_write_kb":28384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"37'7148","reported_seq":7337,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:51.782787+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:51.782787+0000","last_peered":"2024-09-18T11:56:51.782787+0000","last_clean":"2024-09-18T11:56:51.782787+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:51.782787+0000","last_undegraded":"2024-09-18T11:56:51.782787+0000","last_fullsized":"2024-09-18T11:56:51.782787+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29249536,"num_objects":7141,"num_object_clones":0,"num_object_copies":14282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4313,"num_write_kb":17252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"37'7089","reported_seq":7168,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.812578+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:51.812578+0000","last_peered":"2024-09-18T11:56:51.812578+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:51.812578+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7089,"log_dups_size":0,"ondisk_log_size":7089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29017298,"num_objects":7086,"num_object_clones":0,"num_object_copies":14172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":28339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3559},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"37'7157","reported_seq":7242,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.698958+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:51.698958+0000","last_peered":"2024-09-18T11:56:51.698958+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:51.698958+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7157,"log_dups_size":0,"ondisk_log_size":7157,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29291730,"num_objects":7153,"num_object_clones":0,"num_object_copies":14306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7155,"num_write_kb":28608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3594},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"37'7220","reported_seq":7443,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:56:51.787908+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:51.787908+0000","last_peered":"2024-09-18T11:56:51.787908+0000","last_clean":"2024-09-18T11:56:51.787908+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:51.787908+0000","last_undegraded":"2024-09-18T11:56:51.787908+0000","last_fullsized":"2024-09-18T11:56:51.787908+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7220,"log_dups_size":0,"ondisk_log_size":7220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29564928,"num_objects":7218,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.264218+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.264218+0000","last_peered":"2024-09-18T11:55:04.264218+0000","last_clean":"2024-09-18T11:55:04.264218+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.264218+0000","last_undegraded":"2024-09-18T11:55:04.264218+0000","last_fullsized":"2024-09-18T11:55:04.264218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'7151","reported_seq":13876,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:52.687181+0000","last_change":"2024-09-18T11:55:04.628680+0000","last_active":"2024-09-18T11:56:52.687181+0000","last_peered":"2024-09-18T11:56:52.687181+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:52.687181+0000","last_undegraded":"2024-09-18T11:55:04.286738+0000","last_fullsized":"2024-09-18T11:55:04.246740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7151,"log_dups_size":0,"ondisk_log_size":7151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29263058,"num_objects":7146,"num_object_clones":0,"num_object_copies":14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":193,"num_objects_unfound":0,"num_objects_dirty":7146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7149,"num_write_kb":28580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3289,"num_bytes_recovered":13455360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":193},{"shards":"2","objects":6952}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.252384+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:55:03.252384+0000","last_peered":"2024-09-18T11:55:03.252384+0000","last_clean":"2024-09-18T11:55:03.252384+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:55:03.252384+0000","last_undegraded":"2024-09-18T11:55:03.252384+0000","last_fullsized":"2024-09-18T11:55:03.252384+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"37'7187","reported_seq":7263,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.814554+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:51.814554+0000","last_peered":"2024-09-18T11:56:51.814554+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:51.814554+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7187,"log_dups_size":0,"ondisk_log_size":7187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29406418,"num_objects":7181,"num_object_clones":0,"num_object_copies":14362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7185,"num_write_kb":28720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3649},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":89,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.715438+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.715438+0000","last_peered":"2024-09-18T11:55:04.715438+0000","last_clean":"2024-09-18T11:55:04.715438+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.715438+0000","last_undegraded":"2024-09-18T11:55:04.715438+0000","last_fullsized":"2024-09-18T11:55:04.715438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"37'7213","reported_seq":7267,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:52.667169+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:52.667169+0000","last_peered":"2024-09-18T11:56:52.667169+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:52.667169+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29525202,"num_objects":7210,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7211,"num_write_kb":28835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3660}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":69,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.343861+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.343861+0000","last_peered":"2024-09-18T11:55:04.343861+0000","last_clean":"2024-09-18T11:55:04.343861+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.343861+0000","last_undegraded":"2024-09-18T11:55:04.343861+0000","last_fullsized":"2024-09-18T11:55:04.343861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'7154","reported_seq":7361,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:52.635512+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:52.635512+0000","last_peered":"2024-09-18T11:56:52.635512+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:52.635512+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7154,"log_dups_size":0,"ondisk_log_size":7154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29278208,"num_objects":7148,"num_object_clones":0,"num_object_copies":14296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3634,"num_write_kb":14536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"37'7106","reported_seq":7185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.755944+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:51.755944+0000","last_peered":"2024-09-18T11:56:51.755944+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:51.755944+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7106,"log_dups_size":0,"ondisk_log_size":7106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29098601,"num_objects":7105,"num_object_clones":0,"num_object_copies":14210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":28417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3570},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.451721+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.451721+0000","last_peered":"2024-09-18T11:55:04.451721+0000","last_clean":"2024-09-18T11:55:04.451721+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.451721+0000","last_undegraded":"2024-09-18T11:55:04.451721+0000","last_fullsized":"2024-09-18T11:55:04.451721+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"37'7225","reported_seq":7307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.733830+0000","last_change":"2024-09-18T11:55:04.689591+0000","last_active":"2024-09-18T11:56:51.733830+0000","last_peered":"2024-09-18T11:56:51.733830+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:51.733830+0000","last_undegraded":"2024-09-18T11:55:04.322591+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29562683,"num_objects":7220,"num_object_clones":0,"num_object_copies":14440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":28872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3698},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T11:55:03.258797+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:55:03.258797+0000","last_peered":"2024-09-18T11:55:03.258797+0000","last_clean":"2024-09-18T11:55:03.258797+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:55:03.258797+0000","last_undegraded":"2024-09-18T11:55:03.258797+0000","last_fullsized":"2024-09-18T11:55:03.258797+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"37'7089","reported_seq":7168,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.795712+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:51.795712+0000","last_peered":"2024-09-18T11:56:51.795712+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:51.795712+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7089,"log_dups_size":0,"ondisk_log_size":7089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29021394,"num_objects":7087,"num_object_clones":0,"num_object_copies":14174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":28342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3494},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":53,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.427803+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.427803+0000","last_peered":"2024-09-18T11:55:04.427803+0000","last_clean":"2024-09-18T11:55:04.427803+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.427803+0000","last_undegraded":"2024-09-18T11:55:04.427803+0000","last_fullsized":"2024-09-18T11:55:04.427803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.256023+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.256023+0000","last_peered":"2024-09-18T11:55:04.256023+0000","last_clean":"2024-09-18T11:55:04.256023+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.256023+0000","last_undegraded":"2024-09-18T11:55:04.256023+0000","last_fullsized":"2024-09-18T11:55:04.256023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"37'7192","reported_seq":7274,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:51.722294+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:51.722294+0000","last_peered":"2024-09-18T11:56:51.722294+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:51.722294+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7192,"log_dups_size":0,"ondisk_log_size":7192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29411748,"num_objects":7184,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":28727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3661},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":469041204,"num_objects":114529,"num_object_clones":0,"num_object_copies":229058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39090,"num_objects_misplaced":193,"num_objects_unfound":0,"num_objects_dirty":114529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98584,"num_write_kb":394216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3309,"num_bytes_recovered":13464169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":951455744,"data_stored":951375727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114591,"ondisk_log_size":114591,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051620,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645076,"kb_used_data":343052,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93726764,"statfs":{"total":96636764160,"available":95976206336,"internally_reserved":0,"allocated":351285248,"data_stored":351072092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,6,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":3.02}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":4.1970000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":0.42899999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855383,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496672,"kb_used_data":186328,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93875168,"statfs":{"total":96636764160,"available":96128172032,"internally_reserved":0,"allocated":190799872,"data_stored":190513514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.217},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.54000000000000004}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.313},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.78500000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697256,"kb_used_data":399456,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93674584,"statfs":{"total":96636764160,"available":95922774016,"internally_reserved":0,"allocated":409042944,"data_stored":408826166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":1.121}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":409407488,"data_stored":409404009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":190787584,"data_stored":190714525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351260672,"data_stored":351257193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:56:57.138 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 151 20 116875 116855 3.02201 1.64453 0.00536368 0.0258033 2024-09-18T11:56:58.138 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 152 20 117200 117180 3.01048 1.26953 0.0190174 0.0259418 2024-09-18T11:56:59.138 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 153 20 117692 117672 3.00337 1.92188 0.0364376 0.0260026 2024-09-18T11:57:00.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:00.138 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 154 20 118175 118155 2.99612 1.88672 0.0250079 0.0260663 2024-09-18T11:57:00.671 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:00.704 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:00.730 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-18T11:56:59.080121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":478429255,"num_objects":116823,"num_object_clones":0,"num_object_copies":233646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":116823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100878,"num_write_kb":403386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3516,"num_bytes_recovered":14307964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116886,"ondisk_log_size":116886,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1858748,"kb_used_data":948572,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281256772,"statfs":{"total":289910292480,"available":288006934528,"internally_reserved":0,"allocated":971337728,"data_stored":970611038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,9,5,15,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18886656,"num_objects":4611,"num_object_clones":0,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3526,"num_objects_misplaced":3032,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4611,"num_write_kb":18444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":2011136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005067"},"pg_stats":[{"pgid":"2.f","version":"38'7351","reported_seq":7421,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.726211+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:56.726211+0000","last_peered":"2024-09-18T11:56:56.726211+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:56.726211+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30109696,"num_objects":7351,"num_object_clones":0,"num_object_copies":14702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":29404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3752},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'7310","reported_seq":7468,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:57.659003+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:57.659003+0000","last_peered":"2024-09-18T11:56:57.659003+0000","last_clean":"2024-09-18T11:56:57.659003+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:57.659003+0000","last_undegraded":"2024-09-18T11:56:57.659003+0000","last_fullsized":"2024-09-18T11:56:57.659003+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7310,"log_dups_size":0,"ondisk_log_size":7310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29917184,"num_objects":7304,"num_object_clones":0,"num_object_copies":14608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4502,"num_write_kb":18008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'7330","reported_seq":7515,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:57.649233+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:57.649233+0000","last_peered":"2024-09-18T11:56:57.649233+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:57.649233+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7330,"log_dups_size":0,"ondisk_log_size":7330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30007296,"num_objects":7326,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4188,"num_write_kb":16752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3790}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7230","reported_seq":7259,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:57.669095+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:57.669095+0000","last_peered":"2024-09-18T11:56:57.669095+0000","last_clean":"2024-09-18T11:56:57.669095+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:57.669095+0000","last_undegraded":"2024-09-18T11:56:57.669095+0000","last_fullsized":"2024-09-18T11:56:57.669095+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7230,"log_dups_size":0,"ondisk_log_size":7230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29614080,"num_objects":7230,"num_object_clones":0,"num_object_copies":14460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7230,"num_write_kb":28920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"38'7305","reported_seq":7495,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.718350+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:56.718350+0000","last_peered":"2024-09-18T11:56:56.718350+0000","last_clean":"2024-09-18T11:56:56.718350+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:56.718350+0000","last_undegraded":"2024-09-18T11:56:56.718350+0000","last_fullsized":"2024-09-18T11:56:56.718350+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7305,"log_dups_size":0,"ondisk_log_size":7305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29892608,"num_objects":7298,"num_object_clones":0,"num_object_copies":14596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4470,"num_write_kb":17880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"38'7243","reported_seq":7323,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.671795+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:56.671795+0000","last_peered":"2024-09-18T11:56:56.671795+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:56.671795+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7243,"log_dups_size":0,"ondisk_log_size":7243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29648082,"num_objects":7240,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7241,"num_write_kb":28955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3713},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"38'7299","reported_seq":7385,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.834428+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:56.834428+0000","last_peered":"2024-09-18T11:56:56.834428+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:56.834428+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29873362,"num_objects":7295,"num_object_clones":0,"num_object_copies":14590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7297,"num_write_kb":29176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3736},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"38'7390","reported_seq":7614,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.838529+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:56.838529+0000","last_peered":"2024-09-18T11:56:56.838529+0000","last_clean":"2024-09-18T11:56:56.838529+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:56.838529+0000","last_undegraded":"2024-09-18T11:56:56.838529+0000","last_fullsized":"2024-09-18T11:56:56.838529+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7390,"log_dups_size":0,"ondisk_log_size":7390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30261248,"num_objects":7388,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3707,"num_write_kb":14828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.265007+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.265007+0000","last_peered":"2024-09-18T11:55:04.265007+0000","last_clean":"2024-09-18T11:55:04.265007+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.265007+0000","last_undegraded":"2024-09-18T11:55:04.265007+0000","last_fullsized":"2024-09-18T11:55:04.265007+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'7270","reported_seq":14390,"reported_epoch":38,"state":"peering","last_fresh":"2024-09-18T11:56:56.484738+0000","last_change":"2024-09-18T11:56:56.469006+0000","last_active":"2024-09-18T11:56:56.468812+0000","last_peered":"2024-09-18T11:56:56.355567+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:56.484738+0000","last_undegraded":"2024-09-18T11:56:56.484738+0000","last_fullsized":"2024-09-18T11:56:56.484738+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7270,"log_dups_size":0,"ondisk_log_size":7270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29750482,"num_objects":7265,"num_object_clones":0,"num_object_copies":14530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":29056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"38'7330","reported_seq":7407,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.774931+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:56.774931+0000","last_peered":"2024-09-18T11:56:56.774931+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:56.774931+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7330,"log_dups_size":0,"ondisk_log_size":7330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29992146,"num_objects":7324,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7328,"num_write_kb":29292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3792},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":90,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.731217+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.731217+0000","last_peered":"2024-09-18T11:55:04.731217+0000","last_clean":"2024-09-18T11:55:04.731217+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.731217+0000","last_undegraded":"2024-09-18T11:55:04.731217+0000","last_fullsized":"2024-09-18T11:55:04.731217+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'7363","reported_seq":7419,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:57.613653+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:57.613653+0000","last_peered":"2024-09-18T11:56:57.613653+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:57.613653+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7363,"log_dups_size":0,"ondisk_log_size":7363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30139602,"num_objects":7360,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":29435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3810}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":70,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.591747+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.591747+0000","last_peered":"2024-09-18T11:55:04.591747+0000","last_clean":"2024-09-18T11:55:04.591747+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.591747+0000","last_undegraded":"2024-09-18T11:55:04.591747+0000","last_fullsized":"2024-09-18T11:55:04.591747+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'7290","reported_seq":7499,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:57.623576+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:57.623576+0000","last_peered":"2024-09-18T11:56:57.623576+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:57.623576+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7290,"log_dups_size":0,"ondisk_log_size":7290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29835264,"num_objects":7284,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3770,"num_write_kb":15080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3691}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"38'7245","reported_seq":7326,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.792930+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:56.792930+0000","last_peered":"2024-09-18T11:56:56.792930+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:56.792930+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7245,"log_dups_size":0,"ondisk_log_size":7245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29667945,"num_objects":7244,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":28973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3709},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.700867+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.700867+0000","last_peered":"2024-09-18T11:55:04.700867+0000","last_clean":"2024-09-18T11:55:04.700867+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.700867+0000","last_undegraded":"2024-09-18T11:55:04.700867+0000","last_fullsized":"2024-09-18T11:55:04.700867+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"38'7366","reported_seq":7473,"reported_epoch":38,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:56:56.842567+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:56:56.842567+0000","last_peered":"2024-09-18T11:56:56.842567+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:56.842567+0000","last_undegraded":"2024-09-18T11:56:56.842567+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7366,"log_dups_size":0,"ondisk_log_size":7366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30140219,"num_objects":7361,"num_object_clones":0,"num_object_copies":14722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":7361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7363,"num_write_kb":29436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":49152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3851},{"shards":"0,2","objects":3510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"38'7225","reported_seq":7305,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.814657+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:56.814657+0000","last_peered":"2024-09-18T11:56:56.814657+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:56.814657+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29578450,"num_objects":7223,"num_object_clones":0,"num_object_copies":14446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7223,"num_write_kb":28886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3630},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.661859+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.661859+0000","last_peered":"2024-09-18T11:55:04.661859+0000","last_clean":"2024-09-18T11:55:04.661859+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.661859+0000","last_undegraded":"2024-09-18T11:55:04.661859+0000","last_fullsized":"2024-09-18T11:55:04.661859+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":65,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.259318+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.259318+0000","last_peered":"2024-09-18T11:55:04.259318+0000","last_clean":"2024-09-18T11:55:04.259318+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.259318+0000","last_undegraded":"2024-09-18T11:55:04.259318+0000","last_fullsized":"2024-09-18T11:55:04.259318+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"38'7336","reported_seq":7420,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.840376+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:56.840376+0000","last_peered":"2024-09-18T11:56:56.840376+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:56.840376+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7336,"log_dups_size":0,"ondisk_log_size":7336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30001572,"num_objects":7328,"num_object_clones":0,"num_object_copies":14656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7333,"num_write_kb":29303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3805},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":478429236,"num_objects":116821,"num_object_clones":0,"num_object_copies":233642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":116821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100876,"num_write_kb":403384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3515,"num_bytes_recovered":14307945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":971489280,"data_stored":971409263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116883,"ondisk_log_size":116883,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051621,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653884,"kb_used_data":347696,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93717956,"statfs":{"total":96636764160,"available":95967186944,"internally_reserved":0,"allocated":356040704,"data_stored":355822349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,2,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.153},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":5.1829999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.219},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":1.2030000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855384,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496472,"kb_used_data":194444,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93875368,"statfs":{"total":96636764160,"available":96128376832,"internally_reserved":0,"allocated":199110656,"data_stored":198819099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.26000000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.23899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247848,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708392,"kb_used_data":406432,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93663448,"statfs":{"total":96636764160,"available":95911370752,"internally_reserved":0,"allocated":416186368,"data_stored":415969590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,0,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":416587776,"data_stored":416584297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198950912,"data_stored":198877853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":355950592,"data_stored":355947113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:00.732 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:01.141 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 155 20 118611 118591 2.98778 1.70312 0.0715969 0.0261395 2024-09-18T11:57:01.425 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:01.433 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:01.460 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-09-18T11:57:01.080580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":478429255,"num_objects":116823,"num_object_clones":0,"num_object_copies":233646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":116823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100878,"num_write_kb":403386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3516,"num_bytes_recovered":14307964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116886,"ondisk_log_size":116886,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1858748,"kb_used_data":948572,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281256772,"statfs":{"total":289910292480,"available":288006934528,"internally_reserved":0,"allocated":971337728,"data_stored":970611038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,9,5,15,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9388032,"num_objects":2292,"num_object_clones":0,"num_object_copies":4584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3526,"num_objects_misplaced":3317,"num_objects_unfound":0,"num_objects_dirty":2292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2292,"num_write_kb":9168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005087"},"pg_stats":[{"pgid":"2.f","version":"38'7351","reported_seq":7421,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.726211+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:56:56.726211+0000","last_peered":"2024-09-18T11:56:56.726211+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:56:56.726211+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30109696,"num_objects":7351,"num_object_clones":0,"num_object_copies":14702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":29404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3752},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'7310","reported_seq":7468,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:57.659003+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:56:57.659003+0000","last_peered":"2024-09-18T11:56:57.659003+0000","last_clean":"2024-09-18T11:56:57.659003+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:56:57.659003+0000","last_undegraded":"2024-09-18T11:56:57.659003+0000","last_fullsized":"2024-09-18T11:56:57.659003+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7310,"log_dups_size":0,"ondisk_log_size":7310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29917184,"num_objects":7304,"num_object_clones":0,"num_object_copies":14608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4502,"num_write_kb":18008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'7330","reported_seq":7515,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:57.649233+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:56:57.649233+0000","last_peered":"2024-09-18T11:56:57.649233+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:56:57.649233+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7330,"log_dups_size":0,"ondisk_log_size":7330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30007296,"num_objects":7326,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4188,"num_write_kb":16752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3790}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7230","reported_seq":7259,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:56:57.669095+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:56:57.669095+0000","last_peered":"2024-09-18T11:56:57.669095+0000","last_clean":"2024-09-18T11:56:57.669095+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:56:57.669095+0000","last_undegraded":"2024-09-18T11:56:57.669095+0000","last_fullsized":"2024-09-18T11:56:57.669095+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7230,"log_dups_size":0,"ondisk_log_size":7230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29614080,"num_objects":7230,"num_object_clones":0,"num_object_copies":14460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7230,"num_write_kb":28920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"38'7305","reported_seq":7495,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.718350+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:56:56.718350+0000","last_peered":"2024-09-18T11:56:56.718350+0000","last_clean":"2024-09-18T11:56:56.718350+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:56:56.718350+0000","last_undegraded":"2024-09-18T11:56:56.718350+0000","last_fullsized":"2024-09-18T11:56:56.718350+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7305,"log_dups_size":0,"ondisk_log_size":7305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29892608,"num_objects":7298,"num_object_clones":0,"num_object_copies":14596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4470,"num_write_kb":17880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"38'7243","reported_seq":7323,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.671795+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:56:56.671795+0000","last_peered":"2024-09-18T11:56:56.671795+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:56:56.671795+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7243,"log_dups_size":0,"ondisk_log_size":7243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29648082,"num_objects":7240,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7241,"num_write_kb":28955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3713},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"38'7299","reported_seq":7385,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.834428+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:56:56.834428+0000","last_peered":"2024-09-18T11:56:56.834428+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:56:56.834428+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29873362,"num_objects":7295,"num_object_clones":0,"num_object_copies":14590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7297,"num_write_kb":29176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3736},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"38'7390","reported_seq":7614,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.838529+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:56:56.838529+0000","last_peered":"2024-09-18T11:56:56.838529+0000","last_clean":"2024-09-18T11:56:56.838529+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:56:56.838529+0000","last_undegraded":"2024-09-18T11:56:56.838529+0000","last_fullsized":"2024-09-18T11:56:56.838529+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7390,"log_dups_size":0,"ondisk_log_size":7390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30261248,"num_objects":7388,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3707,"num_write_kb":14828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.265007+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:55:04.265007+0000","last_peered":"2024-09-18T11:55:04.265007+0000","last_clean":"2024-09-18T11:55:04.265007+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:55:04.265007+0000","last_undegraded":"2024-09-18T11:55:04.265007+0000","last_fullsized":"2024-09-18T11:55:04.265007+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"37'7270","reported_seq":14390,"reported_epoch":38,"state":"peering","last_fresh":"2024-09-18T11:56:56.484738+0000","last_change":"2024-09-18T11:56:56.469006+0000","last_active":"2024-09-18T11:56:56.468812+0000","last_peered":"2024-09-18T11:56:56.355567+0000","last_clean":"2024-09-18T11:55:02.234203+0000","last_became_active":"2024-09-18T11:55:04.603522+0000","last_became_peered":"2024-09-18T11:55:04.603522+0000","last_unstale":"2024-09-18T11:56:56.484738+0000","last_undegraded":"2024-09-18T11:56:56.484738+0000","last_fullsized":"2024-09-18T11:56:56.484738+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7270,"log_dups_size":0,"ondisk_log_size":7270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29750482,"num_objects":7265,"num_object_clones":0,"num_object_copies":14530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":29056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"38'7330","reported_seq":7407,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.774931+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:56:56.774931+0000","last_peered":"2024-09-18T11:56:56.774931+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:56:56.774931+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7330,"log_dups_size":0,"ondisk_log_size":7330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29992146,"num_objects":7324,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7328,"num_write_kb":29292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3792},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":90,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.731217+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:55:04.731217+0000","last_peered":"2024-09-18T11:55:04.731217+0000","last_clean":"2024-09-18T11:55:04.731217+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:55:04.731217+0000","last_undegraded":"2024-09-18T11:55:04.731217+0000","last_fullsized":"2024-09-18T11:55:04.731217+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'7363","reported_seq":7419,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:57.613653+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:56:57.613653+0000","last_peered":"2024-09-18T11:56:57.613653+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:56:57.613653+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7363,"log_dups_size":0,"ondisk_log_size":7363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30139602,"num_objects":7360,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":29435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3810}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":70,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.591747+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:55:04.591747+0000","last_peered":"2024-09-18T11:55:04.591747+0000","last_clean":"2024-09-18T11:55:04.591747+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:55:04.591747+0000","last_undegraded":"2024-09-18T11:55:04.591747+0000","last_fullsized":"2024-09-18T11:55:04.591747+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'7290","reported_seq":7499,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:57.623576+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:56:57.623576+0000","last_peered":"2024-09-18T11:56:57.623576+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:56:57.623576+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7290,"log_dups_size":0,"ondisk_log_size":7290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29835264,"num_objects":7284,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3770,"num_write_kb":15080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3691}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"38'7245","reported_seq":7326,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.792930+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:56:56.792930+0000","last_peered":"2024-09-18T11:56:56.792930+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:56:56.792930+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7245,"log_dups_size":0,"ondisk_log_size":7245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29667945,"num_objects":7244,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":28973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3709},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.700867+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:55:04.700867+0000","last_peered":"2024-09-18T11:55:04.700867+0000","last_clean":"2024-09-18T11:55:04.700867+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:55:04.700867+0000","last_undegraded":"2024-09-18T11:55:04.700867+0000","last_fullsized":"2024-09-18T11:55:04.700867+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"38'7366","reported_seq":7473,"reported_epoch":38,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:56:56.842567+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:56:56.842567+0000","last_peered":"2024-09-18T11:56:56.842567+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:56:56.842567+0000","last_undegraded":"2024-09-18T11:56:56.842567+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7366,"log_dups_size":0,"ondisk_log_size":7366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30140219,"num_objects":7361,"num_object_clones":0,"num_object_copies":14722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":7361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7363,"num_write_kb":29436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":49152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3851},{"shards":"0,2","objects":3510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"38'7225","reported_seq":7305,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.814657+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:56:56.814657+0000","last_peered":"2024-09-18T11:56:56.814657+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:56:56.814657+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29578450,"num_objects":7223,"num_object_clones":0,"num_object_copies":14446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7223,"num_write_kb":28886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3630},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.661859+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:55:04.661859+0000","last_peered":"2024-09-18T11:55:04.661859+0000","last_clean":"2024-09-18T11:55:04.661859+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:55:04.661859+0000","last_undegraded":"2024-09-18T11:55:04.661859+0000","last_fullsized":"2024-09-18T11:55:04.661859+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":65,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T11:55:04.259318+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:55:04.259318+0000","last_peered":"2024-09-18T11:55:04.259318+0000","last_clean":"2024-09-18T11:55:04.259318+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:55:04.259318+0000","last_undegraded":"2024-09-18T11:55:04.259318+0000","last_fullsized":"2024-09-18T11:55:04.259318+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"38'7336","reported_seq":7420,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:56:56.840376+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:56:56.840376+0000","last_peered":"2024-09-18T11:56:56.840376+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:56:56.840376+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7336,"log_dups_size":0,"ondisk_log_size":7336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30001572,"num_objects":7328,"num_object_clones":0,"num_object_copies":14656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7333,"num_write_kb":29303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3805},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":478429236,"num_objects":116821,"num_object_clones":0,"num_object_copies":233642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":116821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100876,"num_write_kb":403384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3515,"num_bytes_recovered":14307945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":971489280,"data_stored":971409263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116883,"ondisk_log_size":116883,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051621,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653884,"kb_used_data":347696,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93717956,"statfs":{"total":96636764160,"available":95967186944,"internally_reserved":0,"allocated":356040704,"data_stored":355822349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,2,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.153},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":5.1829999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.219},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":1.2030000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855384,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496472,"kb_used_data":194444,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93875368,"statfs":{"total":96636764160,"available":96128376832,"internally_reserved":0,"allocated":199110656,"data_stored":198819099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.26000000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.23899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247848,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708392,"kb_used_data":406432,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93663448,"statfs":{"total":96636764160,"available":95911370752,"internally_reserved":0,"allocated":416186368,"data_stored":415969590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,0,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":416587776,"data_stored":416584297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198950912,"data_stored":198877853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":355950592,"data_stored":355947113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:01.461 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:57:01.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "38'7351", 'reported_seq': 7421, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:56.726211+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:56:56.726211+0000', 'last_peered': '2024-09-18T11:56:56.726211+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:56:56.726211+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7351, 'log_dups_size': 0, 'ondisk_log_size': 7351, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30109696, 'num_objects': 7351, 'num_object_clones': 0, 'num_object_copies': 14702, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7351, 'num_write_kb': 29404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3752}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:01.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:57:01.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'7330", 'reported_seq': 7515, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:57.649233+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:56:57.649233+0000', 'last_peered': '2024-09-18T11:56:57.649233+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:56:57.649233+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 7330, 'log_dups_size': 0, 'ondisk_log_size': 7330, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30007296, 'num_objects': 7326, 'num_object_clones': 0, 'num_object_copies': 14652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4188, 'num_write_kb': 16752, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 3790}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:01.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:57:01.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "38'7243", 'reported_seq': 7323, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:56.671795+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:56:56.671795+0000', 'last_peered': '2024-09-18T11:56:56.671795+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:56:56.671795+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7243, 'log_dups_size': 0, 'ondisk_log_size': 7243, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29648082, 'num_objects': 7240, 'num_object_clones': 0, 'num_object_copies': 14480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7241, 'num_write_kb': 28955, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3713}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:01.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:57:01.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "38'7299", 'reported_seq': 7385, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:56.834428+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:56:56.834428+0000', 'last_peered': '2024-09-18T11:56:56.834428+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:56:56.834428+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7299, 'log_dups_size': 0, 'ondisk_log_size': 7299, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29873362, 'num_objects': 7295, 'num_object_clones': 0, 'num_object_copies': 14590, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7297, 'num_write_kb': 29176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3736}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:01.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-18T11:57:01.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "37'7270", 'reported_seq': 14390, 'reported_epoch': 38, 'state': 'peering', 'last_fresh': '2024-09-18T11:56:56.484738+0000', 'last_change': '2024-09-18T11:56:56.469006+0000', 'last_active': '2024-09-18T11:56:56.468812+0000', 'last_peered': '2024-09-18T11:56:56.355567+0000', 'last_clean': '2024-09-18T11:55:02.234203+0000', 'last_became_active': '2024-09-18T11:55:04.603522+0000', 'last_became_peered': '2024-09-18T11:55:04.603522+0000', 'last_unstale': '2024-09-18T11:56:56.484738+0000', 'last_undegraded': '2024-09-18T11:56:56.484738+0000', 'last_fullsized': '2024-09-18T11:56:56.484738+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 2160, 'log_size': 7270, 'log_dups_size': 0, 'ondisk_log_size': 7270, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29750482, 'num_objects': 7265, 'num_object_clones': 0, 'num_object_copies': 14530, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7268, 'num_write_kb': 29056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3483, 'num_bytes_recovered': 14249984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-18T11:57:01.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:57:01.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "38'7330", 'reported_seq': 7407, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:56.774931+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:56:56.774931+0000', 'last_peered': '2024-09-18T11:56:56.774931+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:56:56.774931+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7330, 'log_dups_size': 0, 'ondisk_log_size': 7330, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29992146, 'num_objects': 7324, 'num_object_clones': 0, 'num_object_copies': 14648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7328, 'num_write_kb': 29292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3792}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:01.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:57:01.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'7363", 'reported_seq': 7419, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:57.613653+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:56:57.613653+0000', 'last_peered': '2024-09-18T11:56:57.613653+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:56:57.613653+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7363, 'log_dups_size': 0, 'ondisk_log_size': 7363, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30139602, 'num_objects': 7360, 'num_object_clones': 0, 'num_object_copies': 14720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7361, 'num_write_kb': 29435, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 3810}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:01.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:57:01.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'7290", 'reported_seq': 7499, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:57.623576+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:56:57.623576+0000', 'last_peered': '2024-09-18T11:56:57.623576+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:56:57.623576+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 7290, 'log_dups_size': 0, 'ondisk_log_size': 7290, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29835264, 'num_objects': 7284, 'num_object_clones': 0, 'num_object_copies': 14568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3770, 'num_write_kb': 15080, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 3691}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:01.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:57:01.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "38'7245", 'reported_seq': 7326, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:56.792930+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:56:56.792930+0000', 'last_peered': '2024-09-18T11:56:56.792930+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:56:56.792930+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7245, 'log_dups_size': 0, 'ondisk_log_size': 7245, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29667945, 'num_objects': 7244, 'num_object_clones': 0, 'num_object_copies': 14488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7244, 'num_write_kb': 28973, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3709}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:01.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:57:01.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "38'7366", 'reported_seq': 7473, 'reported_epoch': 38, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:56:56.842567+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:56:56.842567+0000', 'last_peered': '2024-09-18T11:56:56.842567+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:56:56.842567+0000', 'last_undegraded': '2024-09-18T11:56:56.842567+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7366, 'log_dups_size': 0, 'ondisk_log_size': 7366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30140219, 'num_objects': 7361, 'num_object_clones': 0, 'num_object_copies': 14722, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3510, 'num_objects_unfound': 0, 'num_objects_dirty': 7361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7363, 'num_write_kb': 29436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 49152, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3851}, {'shards': '0,2', 'objects': 3510}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:01.463 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:57:01.464 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "38'7225", 'reported_seq': 7305, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:56.814657+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:56:56.814657+0000', 'last_peered': '2024-09-18T11:56:56.814657+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:56:56.814657+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7225, 'log_dups_size': 0, 'ondisk_log_size': 7225, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29578450, 'num_objects': 7223, 'num_object_clones': 0, 'num_object_copies': 14446, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7223, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7223, 'num_write_kb': 28886, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3630}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:01.464 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:57:01.464 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "38'7336", 'reported_seq': 7420, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:56:56.840376+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:56:56.840376+0000', 'last_peered': '2024-09-18T11:56:56.840376+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:56:56.840376+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7336, 'log_dups_size': 0, 'ondisk_log_size': 7336, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30001572, 'num_objects': 7328, 'num_object_clones': 0, 'num_object_copies': 14656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7333, 'num_write_kb': 29303, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3805}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:01.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:02.138 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 156 20 119066 119046 2.98002 1.77734 0.013815 0.0262023 2024-09-18T11:57:02.490 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:02.490 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/233646 objects degraded (15.221%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":222,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":116823,"data_bytes":478429255,"bytes_used":1903357952,"bytes_avail":288006934528,"bytes_total":289910292480,"inactive_pgs_ratio":0.041666667908430099,"degraded_objects":35564,"degraded_total":233646,"degraded_ratio":0.15221317719969527,"misplaced_objects":3510,"misplaced_total":233646,"misplaced_ratio":0.015022726689093758,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":105404,"recovering_keys_per_sec":0,"num_objects_recovered":206,"num_bytes_recovered":843776,"num_keys_recovered":0,"write_bytes_sec":1172758,"read_op_per_sec":0,"write_op_per_sec":286},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:02.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:02.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:03.138 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 157 20 119529 119509 2.97256 1.80859 0.032204 0.0262732 2024-09-18T11:57:03.224 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:03.233 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:03.251 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-18T11:57:03.081080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":487731271,"num_objects":119094,"num_object_clones":0,"num_object_copies":238188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3381,"num_objects_unfound":0,"num_objects_dirty":119094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103149,"num_write_kb":412470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3645,"num_bytes_recovered":14836348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119157,"ondisk_log_size":119157,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1893544,"kb_used_data":966736,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281221976,"statfs":{"total":289910292480,"available":287971303424,"internally_reserved":0,"allocated":989937664,"data_stored":989190115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,4,10,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18690048,"num_objects":4563,"num_object_clones":0,"num_object_copies":9126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3526,"num_objects_misplaced":3188,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4563,"num_write_kb":18252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":335,"num_bytes_recovered":1372160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001985"},"pg_stats":[{"pgid":"2.f","version":"39'7488","reported_seq":7559,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.752771+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:01.752771+0000","last_peered":"2024-09-18T11:57:01.752771+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:01.752771+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7488,"log_dups_size":0,"ondisk_log_size":7488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30670848,"num_objects":7488,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":29952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3889},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'7453","reported_seq":7607,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:02.690682+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:02.690682+0000","last_peered":"2024-09-18T11:57:02.690682+0000","last_clean":"2024-09-18T11:57:02.690682+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:02.690682+0000","last_undegraded":"2024-09-18T11:57:02.690682+0000","last_fullsized":"2024-09-18T11:57:02.690682+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7453,"log_dups_size":0,"ondisk_log_size":7453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30502912,"num_objects":7447,"num_object_clones":0,"num_object_copies":14894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4645,"num_write_kb":18580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'7480","reported_seq":7665,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:02.631286+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:02.631286+0000","last_peered":"2024-09-18T11:57:02.631286+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:02.631286+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30621696,"num_objects":7476,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4338,"num_write_kb":17352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3940}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7378","reported_seq":7407,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:02.662927+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:02.662927+0000","last_peered":"2024-09-18T11:57:02.662927+0000","last_clean":"2024-09-18T11:57:02.662927+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:02.662927+0000","last_undegraded":"2024-09-18T11:57:02.662927+0000","last_fullsized":"2024-09-18T11:57:02.662927+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7378,"log_dups_size":0,"ondisk_log_size":7378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30220288,"num_objects":7378,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7378,"num_write_kb":29512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'7432","reported_seq":7623,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.778953+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:01.778953+0000","last_peered":"2024-09-18T11:57:01.778953+0000","last_clean":"2024-09-18T11:57:01.778953+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:01.778953+0000","last_undegraded":"2024-09-18T11:57:01.778953+0000","last_fullsized":"2024-09-18T11:57:01.778953+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7432,"log_dups_size":0,"ondisk_log_size":7432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30412800,"num_objects":7425,"num_object_clones":0,"num_object_copies":14850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4597,"num_write_kb":18388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'7394","reported_seq":7475,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.870357+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:01.870357+0000","last_peered":"2024-09-18T11:57:01.870357+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:01.870357+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7394,"log_dups_size":0,"ondisk_log_size":7394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30266578,"num_objects":7391,"num_object_clones":0,"num_object_copies":14782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":29559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3864},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'7449","reported_seq":7534,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.823793+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:01.823793+0000","last_peered":"2024-09-18T11:57:01.823793+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:01.823793+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7449,"log_dups_size":0,"ondisk_log_size":7449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30487762,"num_objects":7445,"num_object_clones":0,"num_object_copies":14890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7447,"num_write_kb":29776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3886},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'7534","reported_seq":7759,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.767690+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:01.767690+0000","last_peered":"2024-09-18T11:57:01.767690+0000","last_clean":"2024-09-18T11:57:01.767690+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:01.767690+0000","last_undegraded":"2024-09-18T11:57:01.767690+0000","last_fullsized":"2024-09-18T11:57:01.767690+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7534,"log_dups_size":0,"ondisk_log_size":7534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30851072,"num_objects":7532,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3851,"num_write_kb":15404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7423","reported_seq":14551,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:02.064334+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:02.064334+0000","last_peered":"2024-09-18T11:57:02.064334+0000","last_clean":"2024-09-18T11:57:02.064334+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:02.064334+0000","last_undegraded":"2024-09-18T11:57:02.064334+0000","last_fullsized":"2024-09-18T11:57:02.064334+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30377170,"num_objects":7418,"num_object_clones":0,"num_object_copies":14836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":29668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'7467","reported_seq":7545,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.812919+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:01.812919+0000","last_peered":"2024-09-18T11:57:01.812919+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:01.812919+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7467,"log_dups_size":0,"ondisk_log_size":7467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30553298,"num_objects":7461,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7465,"num_write_kb":29840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3929},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'7530","reported_seq":7586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:02.613279+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:02.613279+0000","last_peered":"2024-09-18T11:57:02.613279+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:02.613279+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7530,"log_dups_size":0,"ondisk_log_size":7530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30823634,"num_objects":7527,"num_object_clones":0,"num_object_copies":15054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7528,"num_write_kb":30103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'7414","reported_seq":7623,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:02.651712+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:02.651712+0000","last_peered":"2024-09-18T11:57:02.651712+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:02.651712+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7414,"log_dups_size":0,"ondisk_log_size":7414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30343168,"num_objects":7408,"num_object_clones":0,"num_object_copies":14816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3894,"num_write_kb":15576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3815}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'7392","reported_seq":7474,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.778611+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:01.778611+0000","last_peered":"2024-09-18T11:57:01.778611+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:01.778611+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7392,"log_dups_size":0,"ondisk_log_size":7392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30270057,"num_objects":7391,"num_object_clones":0,"num_object_copies":14782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7391,"num_write_kb":29561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3856},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'7499","reported_seq":7864,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:01.856585+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:01.856585+0000","last_peered":"2024-09-18T11:57:01.856585+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:01.856585+0000","last_undegraded":"2024-09-18T11:57:01.856585+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30684987,"num_objects":7494,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3381,"num_objects_unfound":0,"num_objects_dirty":7494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7496,"num_write_kb":29968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":577536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4113},{"shards":"0,2","objects":3381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'7359","reported_seq":7440,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.874901+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:01.874901+0000","last_peered":"2024-09-18T11:57:01.874901+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:01.874901+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7359,"log_dups_size":0,"ondisk_log_size":7359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30127314,"num_objects":7357,"num_object_clones":0,"num_object_copies":14714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7357,"num_write_kb":29422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3764},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'7462","reported_seq":7547,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.873898+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:01.873898+0000","last_peered":"2024-09-18T11:57:01.873898+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:01.873898+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7462,"log_dups_size":0,"ondisk_log_size":7462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30517668,"num_objects":7454,"num_object_clones":0,"num_object_copies":14908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7459,"num_write_kb":29807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3931},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":487731252,"num_objects":119092,"num_object_clones":0,"num_object_copies":238184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3381,"num_objects_unfound":0,"num_objects_dirty":119092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103147,"num_write_kb":412468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3644,"num_bytes_recovered":14836329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":990400512,"data_stored":990320495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119154,"ondisk_log_size":119154,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051622,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662504,"kb_used_data":352160,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93709336,"statfs":{"total":96636764160,"available":95958360064,"internally_reserved":0,"allocated":360611840,"data_stored":360388265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.31},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.215}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.249},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":1.6319999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855385,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507792,"kb_used_data":201608,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93864048,"statfs":{"total":96636764160,"available":96116785152,"internally_reserved":0,"allocated":206446592,"data_stored":206149815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.57499999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":4.8010000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.24299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723248,"kb_used_data":412968,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93648592,"statfs":{"total":96636764160,"available":95896158208,"internally_reserved":0,"allocated":422879232,"data_stored":422652035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":423354368,"data_stored":423350889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206348288,"data_stored":206275229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360697856,"data_stored":360694377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:04.138 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 158 20 120000 119980 2.96539 1.83984 0.0109539 0.0263355 2024-09-18T11:57:05.138 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 159 20 120494 120474 2.95887 1.92969 0.0514782 0.0263949 2024-09-18T11:57:06.142 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:2024-09-18T11:57:06.142891+0000 min lat: 0.0021114 max lat: 2.57663 avg lat: 0.0264551 2024-09-18T11:57:06.142 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:57:06.142 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 160 20 120979 120959 2.95216 1.89453 0.046074 0.0264551 2024-09-18T11:57:06.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:06.993 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:06.999 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:07.029 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-18T11:57:05.081628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":487731271,"num_objects":119094,"num_object_clones":0,"num_object_copies":238188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3381,"num_objects_unfound":0,"num_objects_dirty":119094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103149,"num_write_kb":412470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3645,"num_bytes_recovered":14836348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119157,"ondisk_log_size":119157,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1893544,"kb_used_data":966736,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281221976,"statfs":{"total":289910292480,"available":287971303424,"internally_reserved":0,"allocated":989937664,"data_stored":989190115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,4,10,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18690048,"num_objects":4563,"num_object_clones":0,"num_object_copies":9126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3526,"num_objects_misplaced":3188,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4563,"num_write_kb":18252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":335,"num_bytes_recovered":1372160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.616824"},"pg_stats":[{"pgid":"2.f","version":"39'7488","reported_seq":7559,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.752771+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:01.752771+0000","last_peered":"2024-09-18T11:57:01.752771+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:01.752771+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7488,"log_dups_size":0,"ondisk_log_size":7488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30670848,"num_objects":7488,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":29952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3889},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'7453","reported_seq":7607,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:02.690682+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:02.690682+0000","last_peered":"2024-09-18T11:57:02.690682+0000","last_clean":"2024-09-18T11:57:02.690682+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:02.690682+0000","last_undegraded":"2024-09-18T11:57:02.690682+0000","last_fullsized":"2024-09-18T11:57:02.690682+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7453,"log_dups_size":0,"ondisk_log_size":7453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30502912,"num_objects":7447,"num_object_clones":0,"num_object_copies":14894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4645,"num_write_kb":18580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'7480","reported_seq":7665,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:02.631286+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:02.631286+0000","last_peered":"2024-09-18T11:57:02.631286+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:02.631286+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30621696,"num_objects":7476,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4338,"num_write_kb":17352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3940}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7378","reported_seq":7407,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:02.662927+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:02.662927+0000","last_peered":"2024-09-18T11:57:02.662927+0000","last_clean":"2024-09-18T11:57:02.662927+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:02.662927+0000","last_undegraded":"2024-09-18T11:57:02.662927+0000","last_fullsized":"2024-09-18T11:57:02.662927+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7378,"log_dups_size":0,"ondisk_log_size":7378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30220288,"num_objects":7378,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7378,"num_write_kb":29512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'7432","reported_seq":7623,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.778953+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:01.778953+0000","last_peered":"2024-09-18T11:57:01.778953+0000","last_clean":"2024-09-18T11:57:01.778953+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:01.778953+0000","last_undegraded":"2024-09-18T11:57:01.778953+0000","last_fullsized":"2024-09-18T11:57:01.778953+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7432,"log_dups_size":0,"ondisk_log_size":7432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30412800,"num_objects":7425,"num_object_clones":0,"num_object_copies":14850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4597,"num_write_kb":18388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'7394","reported_seq":7475,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.870357+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:01.870357+0000","last_peered":"2024-09-18T11:57:01.870357+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:01.870357+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7394,"log_dups_size":0,"ondisk_log_size":7394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30266578,"num_objects":7391,"num_object_clones":0,"num_object_copies":14782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":29559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3864},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'7449","reported_seq":7534,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.823793+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:01.823793+0000","last_peered":"2024-09-18T11:57:01.823793+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:01.823793+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7449,"log_dups_size":0,"ondisk_log_size":7449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30487762,"num_objects":7445,"num_object_clones":0,"num_object_copies":14890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7447,"num_write_kb":29776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3886},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'7534","reported_seq":7759,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:01.767690+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:01.767690+0000","last_peered":"2024-09-18T11:57:01.767690+0000","last_clean":"2024-09-18T11:57:01.767690+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:01.767690+0000","last_undegraded":"2024-09-18T11:57:01.767690+0000","last_fullsized":"2024-09-18T11:57:01.767690+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7534,"log_dups_size":0,"ondisk_log_size":7534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30851072,"num_objects":7532,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3851,"num_write_kb":15404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7423","reported_seq":14551,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:02.064334+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:02.064334+0000","last_peered":"2024-09-18T11:57:02.064334+0000","last_clean":"2024-09-18T11:57:02.064334+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:02.064334+0000","last_undegraded":"2024-09-18T11:57:02.064334+0000","last_fullsized":"2024-09-18T11:57:02.064334+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30377170,"num_objects":7418,"num_object_clones":0,"num_object_copies":14836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":29668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'7467","reported_seq":7545,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.812919+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:01.812919+0000","last_peered":"2024-09-18T11:57:01.812919+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:01.812919+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7467,"log_dups_size":0,"ondisk_log_size":7467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30553298,"num_objects":7461,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7465,"num_write_kb":29840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3929},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'7530","reported_seq":7586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:02.613279+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:02.613279+0000","last_peered":"2024-09-18T11:57:02.613279+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:02.613279+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7530,"log_dups_size":0,"ondisk_log_size":7530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30823634,"num_objects":7527,"num_object_clones":0,"num_object_copies":15054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7528,"num_write_kb":30103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'7414","reported_seq":7623,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:02.651712+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:02.651712+0000","last_peered":"2024-09-18T11:57:02.651712+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:02.651712+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7414,"log_dups_size":0,"ondisk_log_size":7414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30343168,"num_objects":7408,"num_object_clones":0,"num_object_copies":14816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3894,"num_write_kb":15576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3815}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'7392","reported_seq":7474,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.778611+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:01.778611+0000","last_peered":"2024-09-18T11:57:01.778611+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:01.778611+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7392,"log_dups_size":0,"ondisk_log_size":7392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30270057,"num_objects":7391,"num_object_clones":0,"num_object_copies":14782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7391,"num_write_kb":29561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3856},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'7499","reported_seq":7864,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:01.856585+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:01.856585+0000","last_peered":"2024-09-18T11:57:01.856585+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:01.856585+0000","last_undegraded":"2024-09-18T11:57:01.856585+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30684987,"num_objects":7494,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3381,"num_objects_unfound":0,"num_objects_dirty":7494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7496,"num_write_kb":29968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":577536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4113},{"shards":"0,2","objects":3381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'7359","reported_seq":7440,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.874901+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:01.874901+0000","last_peered":"2024-09-18T11:57:01.874901+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:01.874901+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7359,"log_dups_size":0,"ondisk_log_size":7359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30127314,"num_objects":7357,"num_object_clones":0,"num_object_copies":14714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7357,"num_write_kb":29422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3764},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'7462","reported_seq":7547,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:01.873898+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:01.873898+0000","last_peered":"2024-09-18T11:57:01.873898+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:01.873898+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7462,"log_dups_size":0,"ondisk_log_size":7462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30517668,"num_objects":7454,"num_object_clones":0,"num_object_copies":14908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7459,"num_write_kb":29807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3931},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":487731252,"num_objects":119092,"num_object_clones":0,"num_object_copies":238184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3381,"num_objects_unfound":0,"num_objects_dirty":119092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103147,"num_write_kb":412468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3644,"num_bytes_recovered":14836329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":990400512,"data_stored":990320495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119154,"ondisk_log_size":119154,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051622,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662504,"kb_used_data":352160,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93709336,"statfs":{"total":96636764160,"available":95958360064,"internally_reserved":0,"allocated":360611840,"data_stored":360388265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.31},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.215}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.249},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":1.6319999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855385,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507792,"kb_used_data":201608,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93864048,"statfs":{"total":96636764160,"available":96116785152,"internally_reserved":0,"allocated":206446592,"data_stored":206149815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.57499999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":4.8010000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.24299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723248,"kb_used_data":412968,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93648592,"statfs":{"total":96636764160,"available":95896158208,"internally_reserved":0,"allocated":422879232,"data_stored":422652035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:56:04 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":4.5430000000000001,"5min":4.5430000000000001,"15min":4.5430000000000001},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":423354368,"data_stored":423350889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206348288,"data_stored":206275229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360697856,"data_stored":360694377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:07.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:07.142 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 161 20 121432 121412 2.94482 1.76953 0.0294031 0.0265212 2024-09-18T11:57:07.768 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:07.773 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:07.796 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-18T11:57:07.082077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":493736007,"num_objects":120560,"num_object_clones":0,"num_object_copies":241120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3286,"num_objects_unfound":0,"num_objects_dirty":120560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104615,"num_write_kb":418334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3740,"num_bytes_recovered":15225468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120623,"ondisk_log_size":120623,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1888284,"kb_used_data":973956,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281227236,"statfs":{"total":289910292480,"available":287976689664,"internally_reserved":0,"allocated":997330944,"data_stored":996583395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,5,16,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18210816,"num_objects":4446,"num_object_clones":0,"num_object_copies":8892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-224,"num_objects_unfound":0,"num_objects_dirty":4446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4446,"num_write_kb":17784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002558"},"pg_stats":[{"pgid":"2.f","version":"39'7604","reported_seq":7675,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.720022+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:06.720022+0000","last_peered":"2024-09-18T11:57:06.720022+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:06.720022+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7604,"log_dups_size":0,"ondisk_log_size":7604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31145984,"num_objects":7604,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":30416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4005},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'7453","reported_seq":7607,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:02.690682+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:02.690682+0000","last_peered":"2024-09-18T11:57:02.690682+0000","last_clean":"2024-09-18T11:57:02.690682+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:02.690682+0000","last_undegraded":"2024-09-18T11:57:02.690682+0000","last_fullsized":"2024-09-18T11:57:02.690682+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7453,"log_dups_size":0,"ondisk_log_size":7453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30502912,"num_objects":7447,"num_object_clones":0,"num_object_copies":14894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4645,"num_write_kb":18580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'7480","reported_seq":7665,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:02.631286+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:02.631286+0000","last_peered":"2024-09-18T11:57:02.631286+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:02.631286+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30621696,"num_objects":7476,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4338,"num_write_kb":17352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":3940}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7378","reported_seq":7407,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:02.662927+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:02.662927+0000","last_peered":"2024-09-18T11:57:02.662927+0000","last_clean":"2024-09-18T11:57:02.662927+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:02.662927+0000","last_undegraded":"2024-09-18T11:57:02.662927+0000","last_fullsized":"2024-09-18T11:57:02.662927+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7378,"log_dups_size":0,"ondisk_log_size":7378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30220288,"num_objects":7378,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7378,"num_write_kb":29512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'7586","reported_seq":7777,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:06.891912+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:06.891912+0000","last_peered":"2024-09-18T11:57:06.891912+0000","last_clean":"2024-09-18T11:57:06.891912+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:06.891912+0000","last_undegraded":"2024-09-18T11:57:06.891912+0000","last_fullsized":"2024-09-18T11:57:06.891912+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7586,"log_dups_size":0,"ondisk_log_size":7586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31043584,"num_objects":7579,"num_object_clones":0,"num_object_copies":15158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4751,"num_write_kb":19004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'7542","reported_seq":7623,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.869818+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:06.869818+0000","last_peered":"2024-09-18T11:57:06.869818+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:06.869818+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7542,"log_dups_size":0,"ondisk_log_size":7542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30872786,"num_objects":7539,"num_object_clones":0,"num_object_copies":15078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":30151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4012},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'7585","reported_seq":7672,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.783945+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:06.783945+0000","last_peered":"2024-09-18T11:57:06.783945+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:06.783945+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7585,"log_dups_size":0,"ondisk_log_size":7585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31044818,"num_objects":7581,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":30320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4022},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'7670","reported_seq":7895,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:06.903047+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:06.903047+0000","last_peered":"2024-09-18T11:57:06.903047+0000","last_clean":"2024-09-18T11:57:06.903047+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:06.903047+0000","last_undegraded":"2024-09-18T11:57:06.903047+0000","last_fullsized":"2024-09-18T11:57:06.903047+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31408128,"num_objects":7668,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":15948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7423","reported_seq":14551,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:02.064334+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:02.064334+0000","last_peered":"2024-09-18T11:57:02.064334+0000","last_clean":"2024-09-18T11:57:02.064334+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:02.064334+0000","last_undegraded":"2024-09-18T11:57:02.064334+0000","last_fullsized":"2024-09-18T11:57:02.064334+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30377170,"num_objects":7418,"num_object_clones":0,"num_object_copies":14836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":29668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'7601","reported_seq":7679,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.850198+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:06.850198+0000","last_peered":"2024-09-18T11:57:06.850198+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:06.850198+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7601,"log_dups_size":0,"ondisk_log_size":7601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31102162,"num_objects":7595,"num_object_clones":0,"num_object_copies":15190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7599,"num_write_kb":30376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4063},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'7530","reported_seq":7586,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:02.613279+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:02.613279+0000","last_peered":"2024-09-18T11:57:02.613279+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:02.613279+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7530,"log_dups_size":0,"ondisk_log_size":7530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30823634,"num_objects":7527,"num_object_clones":0,"num_object_copies":15054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7528,"num_write_kb":30103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":3977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'7414","reported_seq":7623,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:02.651712+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:02.651712+0000","last_peered":"2024-09-18T11:57:02.651712+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:02.651712+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7414,"log_dups_size":0,"ondisk_log_size":7414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30343168,"num_objects":7408,"num_object_clones":0,"num_object_copies":14816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3894,"num_write_kb":15576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3815}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'7544","reported_seq":7626,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.882875+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:06.882875+0000","last_peered":"2024-09-18T11:57:06.882875+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:06.882875+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30892649,"num_objects":7543,"num_object_clones":0,"num_object_copies":15086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7543,"num_write_kb":30169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4008},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'7651","reported_seq":8208,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:06.927564+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:06.927564+0000","last_peered":"2024-09-18T11:57:06.927564+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:06.927564+0000","last_undegraded":"2024-09-18T11:57:06.927564+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7651,"log_dups_size":0,"ondisk_log_size":7651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31307579,"num_objects":7646,"num_object_clones":0,"num_object_copies":15292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3286,"num_objects_unfound":0,"num_objects_dirty":7646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7648,"num_write_kb":30576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":239,"num_bytes_recovered":966656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4360},{"shards":"0,2","objects":3286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'7524","reported_seq":7605,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.903530+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:06.903530+0000","last_peered":"2024-09-18T11:57:06.903530+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:06.903530+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7524,"log_dups_size":0,"ondisk_log_size":7524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30803154,"num_objects":7522,"num_object_clones":0,"num_object_copies":15044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7522,"num_write_kb":30082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3929},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'7635","reported_seq":7720,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.880743+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:06.880743+0000","last_peered":"2024-09-18T11:57:06.880743+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:06.880743+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7635,"log_dups_size":0,"ondisk_log_size":7635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31226276,"num_objects":7627,"num_object_clones":0,"num_object_copies":15254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7632,"num_write_kb":30499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4104},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":493735988,"num_objects":120558,"num_object_clones":0,"num_object_copies":241116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3286,"num_objects_unfound":0,"num_objects_dirty":120558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104613,"num_write_kb":418332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3739,"num_bytes_recovered":15225449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":997711872,"data_stored":997631855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120620,"ondisk_log_size":120620,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051622,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662504,"kb_used_data":352160,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93709336,"statfs":{"total":96636764160,"available":95958360064,"internally_reserved":0,"allocated":360611840,"data_stored":360388265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.31},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.215}]},{"osd":1,"last update":"Wed Sep 18 11:56:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.4080000000000004,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.249},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.984,"5min":3.984,"15min":3.984},"last":1.6319999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855385,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507792,"kb_used_data":201608,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93864048,"statfs":{"total":96636764160,"available":96116785152,"internally_reserved":0,"allocated":206446592,"data_stored":206149815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.7280000000000002,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7779999999999996,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.57499999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4,"5min":4,"15min":4},"last":4.8010000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.24299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717988,"kb_used_data":420188,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93653852,"statfs":{"total":96636764160,"available":95901544448,"internally_reserved":0,"allocated":430272512,"data_stored":430045315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.22600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.38600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":430665728,"data_stored":430662249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206348288,"data_stored":206275229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360697856,"data_stored":360694377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:07.797 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:57:07.798 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'7604", 'reported_seq': 7675, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:06.720022+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:57:06.720022+0000', 'last_peered': '2024-09-18T11:57:06.720022+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:57:06.720022+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7604, 'log_dups_size': 0, 'ondisk_log_size': 7604, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31145984, 'num_objects': 7604, 'num_object_clones': 0, 'num_object_copies': 15208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7604, 'num_write_kb': 30416, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4005}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:07.798 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:57:07.798 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'7480", 'reported_seq': 7665, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:02.631286+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:57:02.631286+0000', 'last_peered': '2024-09-18T11:57:02.631286+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:57:02.631286+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 7480, 'log_dups_size': 0, 'ondisk_log_size': 7480, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30621696, 'num_objects': 7476, 'num_object_clones': 0, 'num_object_copies': 14952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7476, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4338, 'num_write_kb': 17352, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 3940}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:07.798 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:57:07.798 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'7542", 'reported_seq': 7623, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:06.869818+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:57:06.869818+0000', 'last_peered': '2024-09-18T11:57:06.869818+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:57:06.869818+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7542, 'log_dups_size': 0, 'ondisk_log_size': 7542, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30872786, 'num_objects': 7539, 'num_object_clones': 0, 'num_object_copies': 15078, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7540, 'num_write_kb': 30151, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4012}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:07.798 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:57:07.798 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'7585", 'reported_seq': 7672, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:06.783945+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:57:06.783945+0000', 'last_peered': '2024-09-18T11:57:06.783945+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:57:06.783945+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7585, 'log_dups_size': 0, 'ondisk_log_size': 7585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31044818, 'num_objects': 7581, 'num_object_clones': 0, 'num_object_copies': 15162, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7583, 'num_write_kb': 30320, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4022}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:07.798 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:57:07.798 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'7601", 'reported_seq': 7679, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:06.850198+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:57:06.850198+0000', 'last_peered': '2024-09-18T11:57:06.850198+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:57:06.850198+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7601, 'log_dups_size': 0, 'ondisk_log_size': 7601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31102162, 'num_objects': 7595, 'num_object_clones': 0, 'num_object_copies': 15190, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7595, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7599, 'num_write_kb': 30376, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4063}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:07.798 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:57:07.799 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'7530", 'reported_seq': 7586, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:02.613279+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:57:02.613279+0000', 'last_peered': '2024-09-18T11:57:02.613279+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:57:02.613279+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7530, 'log_dups_size': 0, 'ondisk_log_size': 7530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30823634, 'num_objects': 7527, 'num_object_clones': 0, 'num_object_copies': 15054, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7528, 'num_write_kb': 30103, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 3977}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:07.799 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:57:07.799 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'7414", 'reported_seq': 7623, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:02.651712+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:57:02.651712+0000', 'last_peered': '2024-09-18T11:57:02.651712+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:57:02.651712+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 7414, 'log_dups_size': 0, 'ondisk_log_size': 7414, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30343168, 'num_objects': 7408, 'num_object_clones': 0, 'num_object_copies': 14816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3894, 'num_write_kb': 15576, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 3815}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:07.799 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:57:07.799 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'7544", 'reported_seq': 7626, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:06.882875+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:57:06.882875+0000', 'last_peered': '2024-09-18T11:57:06.882875+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:57:06.882875+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7544, 'log_dups_size': 0, 'ondisk_log_size': 7544, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30892649, 'num_objects': 7543, 'num_object_clones': 0, 'num_object_copies': 15086, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7543, 'num_write_kb': 30169, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4008}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:07.799 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:57:07.799 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'7651", 'reported_seq': 8208, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:57:06.927564+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:57:06.927564+0000', 'last_peered': '2024-09-18T11:57:06.927564+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:57:06.927564+0000', 'last_undegraded': '2024-09-18T11:57:06.927564+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7651, 'log_dups_size': 0, 'ondisk_log_size': 7651, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31307579, 'num_objects': 7646, 'num_object_clones': 0, 'num_object_copies': 15292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3286, 'num_objects_unfound': 0, 'num_objects_dirty': 7646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7648, 'num_write_kb': 30576, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 239, 'num_bytes_recovered': 966656, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4360}, {'shards': '0,2', 'objects': 3286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:07.799 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:57:07.800 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'7524", 'reported_seq': 7605, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:06.903530+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:57:06.903530+0000', 'last_peered': '2024-09-18T11:57:06.903530+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:57:06.903530+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7524, 'log_dups_size': 0, 'ondisk_log_size': 7524, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30803154, 'num_objects': 7522, 'num_object_clones': 0, 'num_object_copies': 15044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7522, 'num_write_kb': 30082, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3929}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:07.800 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:57:07.800 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'7635", 'reported_seq': 7720, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:06.880743+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:57:06.880743+0000', 'last_peered': '2024-09-18T11:57:06.880743+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:57:06.880743+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7635, 'log_dups_size': 0, 'ondisk_log_size': 7635, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31226276, 'num_objects': 7627, 'num_object_clones': 0, 'num_object_copies': 15254, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7632, 'num_write_kb': 30499, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4104}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:07.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:08.142 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 162 20 121889 121869 2.93766 1.78516 0.0232643 0.0265857 2024-09-18T11:57:08.642 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:08.642 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/241120 objects degraded (14.750%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":228,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":120560,"data_bytes":493736007,"bytes_used":1933602816,"bytes_avail":287976689664,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":241120,"degraded_ratio":0.14749502322495023,"misplaced_objects":3286,"misplaced_total":241120,"misplaced_ratio":0.01362806901128069,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":91726,"recovering_keys_per_sec":0,"num_objects_recovered":224,"num_bytes_recovered":917504,"num_keys_recovered":0,"write_bytes_sec":1820615,"read_op_per_sec":0,"write_op_per_sec":444},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:08.689 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:08.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:09.143 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 163 19 122337 122318 2.93037 1.75391 0.03245 0.0266536 2024-09-18T11:57:09.401 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:09.409 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:09.442 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-18T11:57:09.082632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":497492039,"num_objects":121477,"num_object_clones":0,"num_object_copies":242954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3286,"num_objects_unfound":0,"num_objects_dirty":121477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105532,"num_write_kb":422002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3740,"num_bytes_recovered":15225468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121540,"ondisk_log_size":121540,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1879516,"kb_used_data":985988,"kb_used_omap":13,"kb_used_meta":893490,"kb_avail":281236004,"statfs":{"total":289910292480,"available":287985668096,"internally_reserved":0,"allocated":1009651712,"data_stored":1008904163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":914934290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,6,12,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21966848,"num_objects":5363,"num_object_clones":0,"num_object_copies":10726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-224,"num_objects_unfound":0,"num_objects_dirty":5363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5363,"num_write_kb":21452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.595680"},"pg_stats":[{"pgid":"2.f","version":"39'7604","reported_seq":7675,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.720022+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:06.720022+0000","last_peered":"2024-09-18T11:57:06.720022+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:06.720022+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7604,"log_dups_size":0,"ondisk_log_size":7604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31145984,"num_objects":7604,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":30416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4005},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'7598","reported_seq":7755,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:07.668510+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:07.668510+0000","last_peered":"2024-09-18T11:57:07.668510+0000","last_clean":"2024-09-18T11:57:07.668510+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:07.668510+0000","last_undegraded":"2024-09-18T11:57:07.668510+0000","last_fullsized":"2024-09-18T11:57:07.668510+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7598,"log_dups_size":0,"ondisk_log_size":7598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31096832,"num_objects":7592,"num_object_clones":0,"num_object_copies":15184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4790,"num_write_kb":19160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'7636","reported_seq":7821,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:07.729233+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:07.729233+0000","last_peered":"2024-09-18T11:57:07.729233+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:07.729233+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7636,"log_dups_size":0,"ondisk_log_size":7636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31260672,"num_objects":7632,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4494,"num_write_kb":17976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4096}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7548","reported_seq":7577,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:07.676904+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:07.676904+0000","last_peered":"2024-09-18T11:57:07.676904+0000","last_clean":"2024-09-18T11:57:07.676904+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:07.676904+0000","last_undegraded":"2024-09-18T11:57:07.676904+0000","last_fullsized":"2024-09-18T11:57:07.676904+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7548,"log_dups_size":0,"ondisk_log_size":7548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30916608,"num_objects":7548,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":30192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'7586","reported_seq":7777,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:06.891912+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:06.891912+0000","last_peered":"2024-09-18T11:57:06.891912+0000","last_clean":"2024-09-18T11:57:06.891912+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:06.891912+0000","last_undegraded":"2024-09-18T11:57:06.891912+0000","last_fullsized":"2024-09-18T11:57:06.891912+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7586,"log_dups_size":0,"ondisk_log_size":7586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31043584,"num_objects":7579,"num_object_clones":0,"num_object_copies":15158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4751,"num_write_kb":19004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'7542","reported_seq":7623,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.869818+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:06.869818+0000","last_peered":"2024-09-18T11:57:06.869818+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:06.869818+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7542,"log_dups_size":0,"ondisk_log_size":7542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30872786,"num_objects":7539,"num_object_clones":0,"num_object_copies":15078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":30151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4012},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'7585","reported_seq":7672,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.783945+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:06.783945+0000","last_peered":"2024-09-18T11:57:06.783945+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:06.783945+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7585,"log_dups_size":0,"ondisk_log_size":7585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31044818,"num_objects":7581,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":30320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4022},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'7670","reported_seq":7895,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:06.903047+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:06.903047+0000","last_peered":"2024-09-18T11:57:06.903047+0000","last_clean":"2024-09-18T11:57:06.903047+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:06.903047+0000","last_undegraded":"2024-09-18T11:57:06.903047+0000","last_fullsized":"2024-09-18T11:57:06.903047+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31408128,"num_objects":7668,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":15948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7569","reported_seq":14697,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:07.058033+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:07.058033+0000","last_peered":"2024-09-18T11:57:07.058033+0000","last_clean":"2024-09-18T11:57:07.058033+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:07.058033+0000","last_undegraded":"2024-09-18T11:57:07.058033+0000","last_fullsized":"2024-09-18T11:57:07.058033+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7569,"log_dups_size":0,"ondisk_log_size":7569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30975186,"num_objects":7564,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7567,"num_write_kb":30252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'7601","reported_seq":7679,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.850198+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:06.850198+0000","last_peered":"2024-09-18T11:57:06.850198+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:06.850198+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7601,"log_dups_size":0,"ondisk_log_size":7601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31102162,"num_objects":7595,"num_object_clones":0,"num_object_copies":15190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7599,"num_write_kb":30376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4063},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'7675","reported_seq":7730,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:07.693695+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:07.693695+0000","last_peered":"2024-09-18T11:57:07.693695+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:07.693695+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7675,"log_dups_size":0,"ondisk_log_size":7675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31417554,"num_objects":7672,"num_object_clones":0,"num_object_copies":15344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7673,"num_write_kb":30683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'7569","reported_seq":7778,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:07.729763+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:07.729763+0000","last_peered":"2024-09-18T11:57:07.729763+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:07.729763+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7569,"log_dups_size":0,"ondisk_log_size":7569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30978048,"num_objects":7563,"num_object_clones":0,"num_object_copies":15126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4049,"num_write_kb":16196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3970}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'7544","reported_seq":7626,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.882875+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:06.882875+0000","last_peered":"2024-09-18T11:57:06.882875+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:06.882875+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30892649,"num_objects":7543,"num_object_clones":0,"num_object_copies":15086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7543,"num_write_kb":30169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4008},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'7651","reported_seq":8208,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:06.927564+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:06.927564+0000","last_peered":"2024-09-18T11:57:06.927564+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:06.927564+0000","last_undegraded":"2024-09-18T11:57:06.927564+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7651,"log_dups_size":0,"ondisk_log_size":7651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31307579,"num_objects":7646,"num_object_clones":0,"num_object_copies":15292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3286,"num_objects_unfound":0,"num_objects_dirty":7646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7648,"num_write_kb":30576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":239,"num_bytes_recovered":966656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4360},{"shards":"0,2","objects":3286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'7524","reported_seq":7605,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.903530+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:06.903530+0000","last_peered":"2024-09-18T11:57:06.903530+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:06.903530+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7524,"log_dups_size":0,"ondisk_log_size":7524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30803154,"num_objects":7522,"num_object_clones":0,"num_object_copies":15044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7522,"num_write_kb":30082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3929},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'7635","reported_seq":7720,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.880743+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:06.880743+0000","last_peered":"2024-09-18T11:57:06.880743+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:06.880743+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7635,"log_dups_size":0,"ondisk_log_size":7635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31226276,"num_objects":7627,"num_object_clones":0,"num_object_copies":15254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7632,"num_write_kb":30499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4104},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":497492020,"num_objects":121475,"num_object_clones":0,"num_object_copies":242950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3286,"num_objects_unfound":0,"num_objects_dirty":121475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105530,"num_write_kb":422000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3739,"num_bytes_recovered":15225449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1010319360,"data_stored":1010239343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121537,"ondisk_log_size":121537,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051623,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654736,"kb_used_data":356872,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93717104,"statfs":{"total":96636764160,"available":95966314496,"internally_reserved":0,"allocated":365436928,"data_stored":365213353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.311}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.34799999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855386,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506792,"kb_used_data":208928,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93865048,"statfs":{"total":96636764160,"available":96117809152,"internally_reserved":0,"allocated":213942272,"data_stored":213645495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.22},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":2.153}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.28000000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717988,"kb_used_data":420188,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93653852,"statfs":{"total":96636764160,"available":95901544448,"internally_reserved":0,"allocated":430272512,"data_stored":430045315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.22600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.38600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":430665728,"data_stored":430662249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213995520,"data_stored":213922461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365658112,"data_stored":365654633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:10.144 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 164 20 122809 122789 2.92373 1.83984 0.0356421 0.0267112 2024-09-18T11:57:11.144 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 165 20 123283 123263 2.91723 1.85156 0.0334975 0.0267718 2024-09-18T11:57:12.144 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 166 20 123762 123742 2.91093 1.87109 0.0638608 0.0268305 2024-09-18T11:57:12.446 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:13.073 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:13.087 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:13.108 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-18T11:57:11.083136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":497492039,"num_objects":121477,"num_object_clones":0,"num_object_copies":242954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3286,"num_objects_unfound":0,"num_objects_dirty":121477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105532,"num_write_kb":422002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3740,"num_bytes_recovered":15225468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121540,"ondisk_log_size":121540,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1879516,"kb_used_data":985988,"kb_used_omap":13,"kb_used_meta":893490,"kb_avail":281236004,"statfs":{"total":289910292480,"available":287985668096,"internally_reserved":0,"allocated":1009651712,"data_stored":1008904163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":914934290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,3,6,12,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19062784,"num_objects":4654,"num_object_clones":0,"num_object_copies":9308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-224,"num_objects_unfound":0,"num_objects_dirty":4654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4654,"num_write_kb":18616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003015"},"pg_stats":[{"pgid":"2.f","version":"39'7604","reported_seq":7675,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.720022+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:06.720022+0000","last_peered":"2024-09-18T11:57:06.720022+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:06.720022+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7604,"log_dups_size":0,"ondisk_log_size":7604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31145984,"num_objects":7604,"num_object_clones":0,"num_object_copies":15208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":30416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4005},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'7598","reported_seq":7755,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:07.668510+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:07.668510+0000","last_peered":"2024-09-18T11:57:07.668510+0000","last_clean":"2024-09-18T11:57:07.668510+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:07.668510+0000","last_undegraded":"2024-09-18T11:57:07.668510+0000","last_fullsized":"2024-09-18T11:57:07.668510+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7598,"log_dups_size":0,"ondisk_log_size":7598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31096832,"num_objects":7592,"num_object_clones":0,"num_object_copies":15184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4790,"num_write_kb":19160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'7636","reported_seq":7821,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:07.729233+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:07.729233+0000","last_peered":"2024-09-18T11:57:07.729233+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:07.729233+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7636,"log_dups_size":0,"ondisk_log_size":7636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31260672,"num_objects":7632,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4494,"num_write_kb":17976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4096}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7548","reported_seq":7577,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:07.676904+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:07.676904+0000","last_peered":"2024-09-18T11:57:07.676904+0000","last_clean":"2024-09-18T11:57:07.676904+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:07.676904+0000","last_undegraded":"2024-09-18T11:57:07.676904+0000","last_fullsized":"2024-09-18T11:57:07.676904+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7548,"log_dups_size":0,"ondisk_log_size":7548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30916608,"num_objects":7548,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":30192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'7586","reported_seq":7777,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:06.891912+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:06.891912+0000","last_peered":"2024-09-18T11:57:06.891912+0000","last_clean":"2024-09-18T11:57:06.891912+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:06.891912+0000","last_undegraded":"2024-09-18T11:57:06.891912+0000","last_fullsized":"2024-09-18T11:57:06.891912+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7586,"log_dups_size":0,"ondisk_log_size":7586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31043584,"num_objects":7579,"num_object_clones":0,"num_object_copies":15158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4751,"num_write_kb":19004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'7542","reported_seq":7623,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.869818+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:06.869818+0000","last_peered":"2024-09-18T11:57:06.869818+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:06.869818+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7542,"log_dups_size":0,"ondisk_log_size":7542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30872786,"num_objects":7539,"num_object_clones":0,"num_object_copies":15078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":30151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4012},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'7585","reported_seq":7672,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.783945+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:06.783945+0000","last_peered":"2024-09-18T11:57:06.783945+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:06.783945+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7585,"log_dups_size":0,"ondisk_log_size":7585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31044818,"num_objects":7581,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":30320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4022},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'7670","reported_seq":7895,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:06.903047+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:06.903047+0000","last_peered":"2024-09-18T11:57:06.903047+0000","last_clean":"2024-09-18T11:57:06.903047+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:06.903047+0000","last_undegraded":"2024-09-18T11:57:06.903047+0000","last_fullsized":"2024-09-18T11:57:06.903047+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31408128,"num_objects":7668,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":15948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7569","reported_seq":14697,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:07.058033+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:07.058033+0000","last_peered":"2024-09-18T11:57:07.058033+0000","last_clean":"2024-09-18T11:57:07.058033+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:07.058033+0000","last_undegraded":"2024-09-18T11:57:07.058033+0000","last_fullsized":"2024-09-18T11:57:07.058033+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7569,"log_dups_size":0,"ondisk_log_size":7569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30975186,"num_objects":7564,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7567,"num_write_kb":30252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'7601","reported_seq":7679,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.850198+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:06.850198+0000","last_peered":"2024-09-18T11:57:06.850198+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:06.850198+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7601,"log_dups_size":0,"ondisk_log_size":7601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31102162,"num_objects":7595,"num_object_clones":0,"num_object_copies":15190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7599,"num_write_kb":30376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4063},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'7675","reported_seq":7730,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:07.693695+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:07.693695+0000","last_peered":"2024-09-18T11:57:07.693695+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:07.693695+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7675,"log_dups_size":0,"ondisk_log_size":7675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31417554,"num_objects":7672,"num_object_clones":0,"num_object_copies":15344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7673,"num_write_kb":30683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'7569","reported_seq":7778,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:07.729763+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:07.729763+0000","last_peered":"2024-09-18T11:57:07.729763+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:07.729763+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7569,"log_dups_size":0,"ondisk_log_size":7569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30978048,"num_objects":7563,"num_object_clones":0,"num_object_copies":15126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4049,"num_write_kb":16196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":3970}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'7544","reported_seq":7626,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.882875+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:06.882875+0000","last_peered":"2024-09-18T11:57:06.882875+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:06.882875+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30892649,"num_objects":7543,"num_object_clones":0,"num_object_copies":15086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7543,"num_write_kb":30169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4008},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'7651","reported_seq":8208,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:06.927564+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:06.927564+0000","last_peered":"2024-09-18T11:57:06.927564+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:06.927564+0000","last_undegraded":"2024-09-18T11:57:06.927564+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7651,"log_dups_size":0,"ondisk_log_size":7651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31307579,"num_objects":7646,"num_object_clones":0,"num_object_copies":15292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3286,"num_objects_unfound":0,"num_objects_dirty":7646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7648,"num_write_kb":30576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":239,"num_bytes_recovered":966656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4360},{"shards":"0,2","objects":3286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'7524","reported_seq":7605,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.903530+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:06.903530+0000","last_peered":"2024-09-18T11:57:06.903530+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:06.903530+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7524,"log_dups_size":0,"ondisk_log_size":7524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30803154,"num_objects":7522,"num_object_clones":0,"num_object_copies":15044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7522,"num_write_kb":30082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3929},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'7635","reported_seq":7720,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:06.880743+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:06.880743+0000","last_peered":"2024-09-18T11:57:06.880743+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:06.880743+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7635,"log_dups_size":0,"ondisk_log_size":7635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31226276,"num_objects":7627,"num_object_clones":0,"num_object_copies":15254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7632,"num_write_kb":30499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4104},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":497492020,"num_objects":121475,"num_object_clones":0,"num_object_copies":242950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3286,"num_objects_unfound":0,"num_objects_dirty":121475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105530,"num_write_kb":422000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3739,"num_bytes_recovered":15225449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1010319360,"data_stored":1010239343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121537,"ondisk_log_size":121537,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051623,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654736,"kb_used_data":356872,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93717104,"statfs":{"total":96636764160,"available":95966314496,"internally_reserved":0,"allocated":365436928,"data_stored":365213353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.311}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.34799999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855386,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506792,"kb_used_data":208928,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93865048,"statfs":{"total":96636764160,"available":96117809152,"internally_reserved":0,"allocated":213942272,"data_stored":213645495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.22},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":2.153}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.28000000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717988,"kb_used_data":420188,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93653852,"statfs":{"total":96636764160,"available":95901544448,"internally_reserved":0,"allocated":430272512,"data_stored":430045315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.22600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.38600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":430665728,"data_stored":430662249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213995520,"data_stored":213922461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365658112,"data_stored":365654633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:13.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:13.144 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 167 20 124218 124198 2.90416 1.78125 0.00869125 0.0268908 2024-09-18T11:57:13.722 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:13.738 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:13.779 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-18T11:57:13.083625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":506982471,"num_objects":123794,"num_object_clones":0,"num_object_copies":247588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3193,"num_objects_unfound":0,"num_objects_dirty":123794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107849,"num_write_kb":431270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3833,"num_bytes_recovered":15606396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123857,"ondisk_log_size":123857,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1910864,"kb_used_data":1004856,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281204656,"statfs":{"total":289910292480,"available":287953567744,"internally_reserved":0,"allocated":1028972544,"data_stored":1028224995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,8,6,13,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28553216,"num_objects":6971,"num_object_clones":0,"num_object_copies":13942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-317,"num_objects_unfound":0,"num_objects_dirty":6971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6971,"num_write_kb":27884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":317,"num_bytes_recovered":1298432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003045"},"pg_stats":[{"pgid":"2.f","version":"39'7756","reported_seq":7827,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.967754+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:11.967754+0000","last_peered":"2024-09-18T11:57:11.967754+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:11.967754+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7756,"log_dups_size":0,"ondisk_log_size":7756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31768576,"num_objects":7756,"num_object_clones":0,"num_object_copies":15512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7756,"num_write_kb":31024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4157},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'7754","reported_seq":7911,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:12.684889+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:12.684889+0000","last_peered":"2024-09-18T11:57:12.684889+0000","last_clean":"2024-09-18T11:57:12.684889+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:12.684889+0000","last_undegraded":"2024-09-18T11:57:12.684889+0000","last_fullsized":"2024-09-18T11:57:12.684889+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7754,"log_dups_size":0,"ondisk_log_size":7754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31735808,"num_objects":7748,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4946,"num_write_kb":19784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'7777","reported_seq":7962,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:12.620721+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:12.620721+0000","last_peered":"2024-09-18T11:57:12.620721+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:12.620721+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31838208,"num_objects":7773,"num_object_clones":0,"num_object_copies":15546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4635,"num_write_kb":18540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4237}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7682","reported_seq":7711,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:12.565307+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:12.565307+0000","last_peered":"2024-09-18T11:57:12.565307+0000","last_clean":"2024-09-18T11:57:12.565307+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:12.565307+0000","last_undegraded":"2024-09-18T11:57:12.565307+0000","last_fullsized":"2024-09-18T11:57:12.565307+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7682,"log_dups_size":0,"ondisk_log_size":7682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31465472,"num_objects":7682,"num_object_clones":0,"num_object_copies":15364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7682,"num_write_kb":30728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'7741","reported_seq":7932,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.850827+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:11.850827+0000","last_peered":"2024-09-18T11:57:11.850827+0000","last_clean":"2024-09-18T11:57:11.850827+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:11.850827+0000","last_undegraded":"2024-09-18T11:57:11.850827+0000","last_fullsized":"2024-09-18T11:57:11.850827+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7741,"log_dups_size":0,"ondisk_log_size":7741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31678464,"num_objects":7734,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4906,"num_write_kb":19624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'7692","reported_seq":7773,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.912989+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:11.912989+0000","last_peered":"2024-09-18T11:57:11.912989+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:11.912989+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31487186,"num_objects":7689,"num_object_clones":0,"num_object_copies":15378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7690,"num_write_kb":30751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4162},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'7716","reported_seq":7803,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.864221+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:11.864221+0000","last_peered":"2024-09-18T11:57:11.864221+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:11.864221+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7716,"log_dups_size":0,"ondisk_log_size":7716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31581394,"num_objects":7712,"num_object_clones":0,"num_object_copies":15424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7714,"num_write_kb":30844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4153},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'7808","reported_seq":8033,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.860025+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:11.860025+0000","last_peered":"2024-09-18T11:57:11.860025+0000","last_clean":"2024-09-18T11:57:11.860025+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:11.860025+0000","last_undegraded":"2024-09-18T11:57:11.860025+0000","last_fullsized":"2024-09-18T11:57:11.860025+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7808,"log_dups_size":0,"ondisk_log_size":7808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31973376,"num_objects":7806,"num_object_clones":0,"num_object_copies":15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":16500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7713","reported_seq":14841,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.979031+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:11.979031+0000","last_peered":"2024-09-18T11:57:11.979031+0000","last_clean":"2024-09-18T11:57:11.979031+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:11.979031+0000","last_undegraded":"2024-09-18T11:57:11.979031+0000","last_fullsized":"2024-09-18T11:57:11.979031+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31565010,"num_objects":7708,"num_object_clones":0,"num_object_copies":15416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7711,"num_write_kb":30828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'7735","reported_seq":7813,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.972196+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:11.972196+0000","last_peered":"2024-09-18T11:57:11.972196+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:11.972196+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7735,"log_dups_size":0,"ondisk_log_size":7735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31651026,"num_objects":7729,"num_object_clones":0,"num_object_copies":15458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7733,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4197},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'7811","reported_seq":7867,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:12.644558+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:12.644558+0000","last_peered":"2024-09-18T11:57:12.644558+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:12.644558+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7811,"log_dups_size":0,"ondisk_log_size":7811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31974610,"num_objects":7808,"num_object_clones":0,"num_object_copies":15616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7809,"num_write_kb":31227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4258}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'7716","reported_seq":7925,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:12.687001+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:12.687001+0000","last_peered":"2024-09-18T11:57:12.687001+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:12.687001+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7716,"log_dups_size":0,"ondisk_log_size":7716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31580160,"num_objects":7710,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4196,"num_write_kb":16784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4117}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'7710","reported_seq":7792,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.938557+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:11.938557+0000","last_peered":"2024-09-18T11:57:11.938557+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:11.938557+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31572585,"num_objects":7709,"num_object_clones":0,"num_object_copies":15418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7709,"num_write_kb":30833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4174},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'7801","reported_seq":8540,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:11.969518+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:11.969518+0000","last_peered":"2024-09-18T11:57:11.969518+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:11.969518+0000","last_undegraded":"2024-09-18T11:57:11.969518+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31921979,"num_objects":7796,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3193,"num_objects_unfound":0,"num_objects_dirty":7796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7798,"num_write_kb":31176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":1347584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4603},{"shards":"0,2","objects":3193}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'7665","reported_seq":7746,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.952638+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:11.952638+0000","last_peered":"2024-09-18T11:57:11.952638+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:11.952638+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7665,"log_dups_size":0,"ondisk_log_size":7665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31380690,"num_objects":7663,"num_object_clones":0,"num_object_copies":15326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7663,"num_write_kb":30646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4070},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'7777","reported_seq":7862,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.856548+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:11.856548+0000","last_peered":"2024-09-18T11:57:11.856548+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:11.856548+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31807908,"num_objects":7769,"num_object_clones":0,"num_object_copies":15538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":31067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4246},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":506982452,"num_objects":123792,"num_object_clones":0,"num_object_copies":247584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3193,"num_objects_unfound":0,"num_objects_dirty":123792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107847,"num_write_kb":431268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3832,"num_bytes_recovered":15606377,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1029779456,"data_stored":1029699439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123854,"ondisk_log_size":123854,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051624,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663560,"kb_used_data":361536,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93708280,"statfs":{"total":96636764160,"available":95957278720,"internally_reserved":0,"allocated":370212864,"data_stored":369989289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.25700000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.31900000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855387,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518244,"kb_used_data":216220,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93853596,"statfs":{"total":96636764160,"available":96106082304,"internally_reserved":0,"allocated":221409280,"data_stored":221112503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.22},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":2.153}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.28000000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729060,"kb_used_data":427100,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93642780,"statfs":{"total":96636764160,"available":95890206720,"internally_reserved":0,"allocated":437350400,"data_stored":437123203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,10,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":4.617}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":4.883},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.253}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":437841920,"data_stored":437838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221536256,"data_stored":221463197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370401280,"data_stored":370397801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:13.781 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:57:13.781 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'7756", 'reported_seq': 7827, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:11.967754+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:57:11.967754+0000', 'last_peered': '2024-09-18T11:57:11.967754+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:57:11.967754+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7756, 'log_dups_size': 0, 'ondisk_log_size': 7756, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31768576, 'num_objects': 7756, 'num_object_clones': 0, 'num_object_copies': 15512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7756, 'num_write_kb': 31024, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4157}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:13.781 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:57:13.781 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'7777", 'reported_seq': 7962, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:12.620721+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:57:12.620721+0000', 'last_peered': '2024-09-18T11:57:12.620721+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:57:12.620721+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 7777, 'log_dups_size': 0, 'ondisk_log_size': 7777, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31838208, 'num_objects': 7773, 'num_object_clones': 0, 'num_object_copies': 15546, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4635, 'num_write_kb': 18540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 4237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:13.781 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:57:13.781 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'7692", 'reported_seq': 7773, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:11.912989+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:57:11.912989+0000', 'last_peered': '2024-09-18T11:57:11.912989+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:57:11.912989+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7692, 'log_dups_size': 0, 'ondisk_log_size': 7692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31487186, 'num_objects': 7689, 'num_object_clones': 0, 'num_object_copies': 15378, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7690, 'num_write_kb': 30751, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4162}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:13.781 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:57:13.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'7716", 'reported_seq': 7803, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:11.864221+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:57:11.864221+0000', 'last_peered': '2024-09-18T11:57:11.864221+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:57:11.864221+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7716, 'log_dups_size': 0, 'ondisk_log_size': 7716, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31581394, 'num_objects': 7712, 'num_object_clones': 0, 'num_object_copies': 15424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7712, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7714, 'num_write_kb': 30844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4153}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:13.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:57:13.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'7735", 'reported_seq': 7813, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:11.972196+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:57:11.972196+0000', 'last_peered': '2024-09-18T11:57:11.972196+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:57:11.972196+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7735, 'log_dups_size': 0, 'ondisk_log_size': 7735, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31651026, 'num_objects': 7729, 'num_object_clones': 0, 'num_object_copies': 15458, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7733, 'num_write_kb': 30912, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4197}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:13.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:57:13.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'7811", 'reported_seq': 7867, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:12.644558+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:57:12.644558+0000', 'last_peered': '2024-09-18T11:57:12.644558+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:57:12.644558+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7811, 'log_dups_size': 0, 'ondisk_log_size': 7811, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31974610, 'num_objects': 7808, 'num_object_clones': 0, 'num_object_copies': 15616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7808, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7809, 'num_write_kb': 31227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 4258}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:13.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:57:13.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'7716", 'reported_seq': 7925, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:12.687001+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:57:12.687001+0000', 'last_peered': '2024-09-18T11:57:12.687001+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:57:12.687001+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 7716, 'log_dups_size': 0, 'ondisk_log_size': 7716, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31580160, 'num_objects': 7710, 'num_object_clones': 0, 'num_object_copies': 15420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4196, 'num_write_kb': 16784, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 4117}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:13.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:57:13.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'7710", 'reported_seq': 7792, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:11.938557+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:57:11.938557+0000', 'last_peered': '2024-09-18T11:57:11.938557+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:57:11.938557+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7710, 'log_dups_size': 0, 'ondisk_log_size': 7710, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31572585, 'num_objects': 7709, 'num_object_clones': 0, 'num_object_copies': 15418, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7709, 'num_write_kb': 30833, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4174}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:13.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:57:13.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'7801", 'reported_seq': 8540, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:57:11.969518+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:57:11.969518+0000', 'last_peered': '2024-09-18T11:57:11.969518+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:57:11.969518+0000', 'last_undegraded': '2024-09-18T11:57:11.969518+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7801, 'log_dups_size': 0, 'ondisk_log_size': 7801, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31921979, 'num_objects': 7796, 'num_object_clones': 0, 'num_object_copies': 15592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3193, 'num_objects_unfound': 0, 'num_objects_dirty': 7796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7798, 'num_write_kb': 31176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 332, 'num_bytes_recovered': 1347584, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4603}, {'shards': '0,2', 'objects': 3193}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:13.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:57:13.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'7665", 'reported_seq': 7746, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:11.952638+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:57:11.952638+0000', 'last_peered': '2024-09-18T11:57:11.952638+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:57:11.952638+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7665, 'log_dups_size': 0, 'ondisk_log_size': 7665, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31380690, 'num_objects': 7663, 'num_object_clones': 0, 'num_object_copies': 15326, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7663, 'num_write_kb': 30646, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4070}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:13.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:57:13.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'7777", 'reported_seq': 7862, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:11.856548+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:57:11.856548+0000', 'last_peered': '2024-09-18T11:57:11.856548+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:57:11.856548+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7777, 'log_dups_size': 0, 'ondisk_log_size': 7777, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31807908, 'num_objects': 7769, 'num_object_clones': 0, 'num_object_copies': 15538, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7774, 'num_write_kb': 31067, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4246}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:13.783 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:14.144 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 168 20 124667 124647 2.89732 1.75391 0.0106861 0.0269568 2024-09-18T11:57:14.649 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:14.649 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/247588 objects degraded (14.364%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":234,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":123794,"data_bytes":506982471,"bytes_used":1956724736,"bytes_avail":287953567744,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":247588,"degraded_ratio":0.14364185663279319,"misplaced_objects":3193,"misplaced_total":247588,"misplaced_ratio":0.012896424705559234,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":108175,"recovering_keys_per_sec":0,"num_objects_recovered":317,"num_bytes_recovered":1298432,"num_keys_recovered":0,"write_bytes_sec":2378830,"read_op_per_sec":0,"write_op_per_sec":580},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:14.681 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:14.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:15.144 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 169 20 125105 125085 2.8903 1.71094 0.0796404 0.0270224 2024-09-18T11:57:15.217 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:15.231 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:15.251 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-18T11:57:15.084124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":506982471,"num_objects":123794,"num_object_clones":0,"num_object_copies":247588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3193,"num_objects_unfound":0,"num_objects_dirty":123794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107849,"num_write_kb":431270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3833,"num_bytes_recovered":15606396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123857,"ondisk_log_size":123857,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1910864,"kb_used_data":1004856,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281204656,"statfs":{"total":289910292480,"available":287953567744,"internally_reserved":0,"allocated":1028972544,"data_stored":1028224995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,8,6,13,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19251200,"num_objects":4700,"num_object_clones":0,"num_object_copies":9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-188,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4700,"num_write_kb":18800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":770048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003043"},"pg_stats":[{"pgid":"2.f","version":"39'7756","reported_seq":7827,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.967754+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:11.967754+0000","last_peered":"2024-09-18T11:57:11.967754+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:11.967754+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7756,"log_dups_size":0,"ondisk_log_size":7756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31768576,"num_objects":7756,"num_object_clones":0,"num_object_copies":15512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7756,"num_write_kb":31024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4157},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'7754","reported_seq":7911,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:12.684889+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:12.684889+0000","last_peered":"2024-09-18T11:57:12.684889+0000","last_clean":"2024-09-18T11:57:12.684889+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:12.684889+0000","last_undegraded":"2024-09-18T11:57:12.684889+0000","last_fullsized":"2024-09-18T11:57:12.684889+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7754,"log_dups_size":0,"ondisk_log_size":7754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31735808,"num_objects":7748,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4946,"num_write_kb":19784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'7777","reported_seq":7962,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:12.620721+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:12.620721+0000","last_peered":"2024-09-18T11:57:12.620721+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:12.620721+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31838208,"num_objects":7773,"num_object_clones":0,"num_object_copies":15546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4635,"num_write_kb":18540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4237}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7682","reported_seq":7711,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:12.565307+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:12.565307+0000","last_peered":"2024-09-18T11:57:12.565307+0000","last_clean":"2024-09-18T11:57:12.565307+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:12.565307+0000","last_undegraded":"2024-09-18T11:57:12.565307+0000","last_fullsized":"2024-09-18T11:57:12.565307+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7682,"log_dups_size":0,"ondisk_log_size":7682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31465472,"num_objects":7682,"num_object_clones":0,"num_object_copies":15364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7682,"num_write_kb":30728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'7741","reported_seq":7932,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.850827+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:11.850827+0000","last_peered":"2024-09-18T11:57:11.850827+0000","last_clean":"2024-09-18T11:57:11.850827+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:11.850827+0000","last_undegraded":"2024-09-18T11:57:11.850827+0000","last_fullsized":"2024-09-18T11:57:11.850827+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7741,"log_dups_size":0,"ondisk_log_size":7741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31678464,"num_objects":7734,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4906,"num_write_kb":19624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'7692","reported_seq":7773,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.912989+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:11.912989+0000","last_peered":"2024-09-18T11:57:11.912989+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:11.912989+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31487186,"num_objects":7689,"num_object_clones":0,"num_object_copies":15378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7690,"num_write_kb":30751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4162},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'7716","reported_seq":7803,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.864221+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:11.864221+0000","last_peered":"2024-09-18T11:57:11.864221+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:11.864221+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7716,"log_dups_size":0,"ondisk_log_size":7716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31581394,"num_objects":7712,"num_object_clones":0,"num_object_copies":15424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7714,"num_write_kb":30844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4153},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'7808","reported_seq":8033,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.860025+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:11.860025+0000","last_peered":"2024-09-18T11:57:11.860025+0000","last_clean":"2024-09-18T11:57:11.860025+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:11.860025+0000","last_undegraded":"2024-09-18T11:57:11.860025+0000","last_fullsized":"2024-09-18T11:57:11.860025+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7808,"log_dups_size":0,"ondisk_log_size":7808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31973376,"num_objects":7806,"num_object_clones":0,"num_object_copies":15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":16500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7713","reported_seq":14841,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.979031+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:11.979031+0000","last_peered":"2024-09-18T11:57:11.979031+0000","last_clean":"2024-09-18T11:57:11.979031+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:11.979031+0000","last_undegraded":"2024-09-18T11:57:11.979031+0000","last_fullsized":"2024-09-18T11:57:11.979031+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31565010,"num_objects":7708,"num_object_clones":0,"num_object_copies":15416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7711,"num_write_kb":30828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'7735","reported_seq":7813,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.972196+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:11.972196+0000","last_peered":"2024-09-18T11:57:11.972196+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:11.972196+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7735,"log_dups_size":0,"ondisk_log_size":7735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31651026,"num_objects":7729,"num_object_clones":0,"num_object_copies":15458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7733,"num_write_kb":30912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4197},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'7811","reported_seq":7867,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:12.644558+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:12.644558+0000","last_peered":"2024-09-18T11:57:12.644558+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:12.644558+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7811,"log_dups_size":0,"ondisk_log_size":7811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31974610,"num_objects":7808,"num_object_clones":0,"num_object_copies":15616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7809,"num_write_kb":31227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4258}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'7716","reported_seq":7925,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:12.687001+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:12.687001+0000","last_peered":"2024-09-18T11:57:12.687001+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:12.687001+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7716,"log_dups_size":0,"ondisk_log_size":7716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31580160,"num_objects":7710,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4196,"num_write_kb":16784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4117}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'7710","reported_seq":7792,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.938557+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:11.938557+0000","last_peered":"2024-09-18T11:57:11.938557+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:11.938557+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31572585,"num_objects":7709,"num_object_clones":0,"num_object_copies":15418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7709,"num_write_kb":30833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4174},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'7801","reported_seq":8540,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:11.969518+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:11.969518+0000","last_peered":"2024-09-18T11:57:11.969518+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:11.969518+0000","last_undegraded":"2024-09-18T11:57:11.969518+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31921979,"num_objects":7796,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3193,"num_objects_unfound":0,"num_objects_dirty":7796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7798,"num_write_kb":31176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":1347584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4603},{"shards":"0,2","objects":3193}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'7665","reported_seq":7746,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.952638+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:11.952638+0000","last_peered":"2024-09-18T11:57:11.952638+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:11.952638+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7665,"log_dups_size":0,"ondisk_log_size":7665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31380690,"num_objects":7663,"num_object_clones":0,"num_object_copies":15326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7663,"num_write_kb":30646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4070},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'7777","reported_seq":7862,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:11.856548+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:11.856548+0000","last_peered":"2024-09-18T11:57:11.856548+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:11.856548+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31807908,"num_objects":7769,"num_object_clones":0,"num_object_copies":15538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":31067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4246},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":506982452,"num_objects":123792,"num_object_clones":0,"num_object_copies":247584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3193,"num_objects_unfound":0,"num_objects_dirty":123792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107847,"num_write_kb":431268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3832,"num_bytes_recovered":15606377,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1029779456,"data_stored":1029699439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123854,"ondisk_log_size":123854,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051624,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663560,"kb_used_data":361536,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93708280,"statfs":{"total":96636764160,"available":95957278720,"internally_reserved":0,"allocated":370212864,"data_stored":369989289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.25700000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.31900000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855387,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518244,"kb_used_data":216220,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93853596,"statfs":{"total":96636764160,"available":96106082304,"internally_reserved":0,"allocated":221409280,"data_stored":221112503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.22},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":2.153}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.28000000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729060,"kb_used_data":427100,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93642780,"statfs":{"total":96636764160,"available":95890206720,"internally_reserved":0,"allocated":437350400,"data_stored":437123203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,10,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":4.617}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":4.883},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":0.253}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":437841920,"data_stored":437838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221536256,"data_stored":221463197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370401280,"data_stored":370397801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:16.144 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 170 20 125585 125565 2.88433 1.875 0.0776688 0.027079 2024-09-18T11:57:17.144 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 171 19 126056 126037 2.87824 1.84375 0.0408821 0.0271336 2024-09-18T11:57:18.144 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 172 20 126528 126508 2.8722 1.83984 0.0554284 0.0271923 2024-09-18T11:57:18.253 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:18.900 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:18.908 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:18.940 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-18T11:57:17.084547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":512917575,"num_objects":125243,"num_object_clones":0,"num_object_copies":250486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3106,"num_objects_unfound":0,"num_objects_dirty":125243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109298,"num_write_kb":437066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3920,"num_bytes_recovered":15962748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125306,"ondisk_log_size":125306,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1926268,"kb_used_data":1011940,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281189252,"statfs":{"total":289910292480,"available":287937794048,"internally_reserved":0,"allocated":1036226560,"data_stored":1035479011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,5,8,9,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25186304,"num_objects":6149,"num_object_clones":0,"num_object_copies":12298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-275,"num_objects_unfound":0,"num_objects_dirty":6149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6149,"num_write_kb":24596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1126400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002919"},"pg_stats":[{"pgid":"2.f","version":"39'7886","reported_seq":7957,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.802525+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:16.802525+0000","last_peered":"2024-09-18T11:57:16.802525+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:16.802525+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7886,"log_dups_size":0,"ondisk_log_size":7886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32301056,"num_objects":7886,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7886,"num_write_kb":31544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4287},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'7754","reported_seq":7911,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:12.684889+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:12.684889+0000","last_peered":"2024-09-18T11:57:12.684889+0000","last_clean":"2024-09-18T11:57:12.684889+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:12.684889+0000","last_undegraded":"2024-09-18T11:57:12.684889+0000","last_fullsized":"2024-09-18T11:57:12.684889+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7754,"log_dups_size":0,"ondisk_log_size":7754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31735808,"num_objects":7748,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4946,"num_write_kb":19784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'7777","reported_seq":7962,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:12.620721+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:12.620721+0000","last_peered":"2024-09-18T11:57:12.620721+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:12.620721+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31838208,"num_objects":7773,"num_object_clones":0,"num_object_copies":15546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4635,"num_write_kb":18540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4237}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7682","reported_seq":7711,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:12.565307+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:12.565307+0000","last_peered":"2024-09-18T11:57:12.565307+0000","last_clean":"2024-09-18T11:57:12.565307+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:12.565307+0000","last_undegraded":"2024-09-18T11:57:12.565307+0000","last_fullsized":"2024-09-18T11:57:12.565307+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7682,"log_dups_size":0,"ondisk_log_size":7682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31465472,"num_objects":7682,"num_object_clones":0,"num_object_copies":15364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7682,"num_write_kb":30728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'7894","reported_seq":8085,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:16.904756+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:16.904756+0000","last_peered":"2024-09-18T11:57:16.904756+0000","last_clean":"2024-09-18T11:57:16.904756+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:16.904756+0000","last_undegraded":"2024-09-18T11:57:16.904756+0000","last_fullsized":"2024-09-18T11:57:16.904756+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7894,"log_dups_size":0,"ondisk_log_size":7894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32305152,"num_objects":7887,"num_object_clones":0,"num_object_copies":15774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5059,"num_write_kb":20236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'7835","reported_seq":7916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.921612+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:16.921612+0000","last_peered":"2024-09-18T11:57:16.921612+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:16.921612+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7835,"log_dups_size":0,"ondisk_log_size":7835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32072914,"num_objects":7832,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7833,"num_write_kb":31323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4305},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'7864","reported_seq":7951,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.941885+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:16.941885+0000","last_peered":"2024-09-18T11:57:16.941885+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:16.941885+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7864,"log_dups_size":0,"ondisk_log_size":7864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32187602,"num_objects":7860,"num_object_clones":0,"num_object_copies":15720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7862,"num_write_kb":31436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4301},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'7974","reported_seq":8199,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:16.975023+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:16.975023+0000","last_peered":"2024-09-18T11:57:16.975023+0000","last_clean":"2024-09-18T11:57:16.975023+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:16.975023+0000","last_undegraded":"2024-09-18T11:57:16.975023+0000","last_fullsized":"2024-09-18T11:57:16.975023+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7974,"log_dups_size":0,"ondisk_log_size":7974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32653312,"num_objects":7972,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4291,"num_write_kb":17164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7713","reported_seq":14841,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:11.979031+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:11.979031+0000","last_peered":"2024-09-18T11:57:11.979031+0000","last_clean":"2024-09-18T11:57:11.979031+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:11.979031+0000","last_undegraded":"2024-09-18T11:57:11.979031+0000","last_fullsized":"2024-09-18T11:57:11.979031+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31565010,"num_objects":7708,"num_object_clones":0,"num_object_copies":15416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7711,"num_write_kb":30828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'7874","reported_seq":7952,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.985385+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:16.985385+0000","last_peered":"2024-09-18T11:57:16.985385+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:16.985385+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7874,"log_dups_size":0,"ondisk_log_size":7874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32220370,"num_objects":7868,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7872,"num_write_kb":31468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4336},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'7811","reported_seq":7867,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:12.644558+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:12.644558+0000","last_peered":"2024-09-18T11:57:12.644558+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:12.644558+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7811,"log_dups_size":0,"ondisk_log_size":7811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31974610,"num_objects":7808,"num_object_clones":0,"num_object_copies":15616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7809,"num_write_kb":31227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4258}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'7716","reported_seq":7925,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:12.687001+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:12.687001+0000","last_peered":"2024-09-18T11:57:12.687001+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:12.687001+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7716,"log_dups_size":0,"ondisk_log_size":7716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31580160,"num_objects":7710,"num_object_clones":0,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4196,"num_write_kb":16784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4117}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'7862","reported_seq":7944,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.896150+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:16.896150+0000","last_peered":"2024-09-18T11:57:16.896150+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:16.896150+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32195177,"num_objects":7861,"num_object_clones":0,"num_object_copies":15722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":31441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4326},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'7944","reported_seq":8860,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:16.976121+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:16.976121+0000","last_peered":"2024-09-18T11:57:16.976121+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:16.976121+0000","last_undegraded":"2024-09-18T11:57:16.976121+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7944,"log_dups_size":0,"ondisk_log_size":7944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32507707,"num_objects":7939,"num_object_clones":0,"num_object_copies":15878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3106,"num_objects_unfound":0,"num_objects_dirty":7939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":31748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":419,"num_bytes_recovered":1703936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4833},{"shards":"0,2","objects":3106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'7805","reported_seq":7886,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.789045+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:16.789045+0000","last_peered":"2024-09-18T11:57:16.789045+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:16.789045+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7805,"log_dups_size":0,"ondisk_log_size":7805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31954130,"num_objects":7803,"num_object_clones":0,"num_object_copies":15606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7803,"num_write_kb":31206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4210},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'7912","reported_seq":7997,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.938252+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:16.938252+0000","last_peered":"2024-09-18T11:57:16.938252+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:16.938252+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7912,"log_dups_size":0,"ondisk_log_size":7912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32360868,"num_objects":7904,"num_object_clones":0,"num_object_copies":15808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7909,"num_write_kb":31607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4381},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":512917556,"num_objects":125241,"num_object_clones":0,"num_object_copies":250482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3106,"num_objects_unfound":0,"num_objects_dirty":125241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109296,"num_write_kb":437064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3919,"num_bytes_recovered":15962729,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1036939264,"data_stored":1036859247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125303,"ondisk_log_size":125303,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051624,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663560,"kb_used_data":361536,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93708280,"statfs":{"total":96636764160,"available":95957278720,"internally_reserved":0,"allocated":370212864,"data_stored":369989289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.25700000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.31900000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855387,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518244,"kb_used_data":216220,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93853596,"statfs":{"total":96636764160,"available":96106082304,"internally_reserved":0,"allocated":221409280,"data_stored":221112503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.22},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":2.153}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.28000000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744464,"kb_used_data":434184,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93627376,"statfs":{"total":96636764160,"available":95874433024,"internally_reserved":0,"allocated":444604416,"data_stored":444377219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":2.7130000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":2.238}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":2.379},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":2.7490000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":445001728,"data_stored":444998249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221536256,"data_stored":221463197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370401280,"data_stored":370397801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:18.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:19.144 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 173 20 126955 126935 2.86524 1.66797 0.0524061 0.0272581 2024-09-18T11:57:19.739 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:19.758 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:19.780 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-18T11:57:19.085069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516419655,"num_objects":126098,"num_object_clones":0,"num_object_copies":252196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3106,"num_objects_unfound":0,"num_objects_dirty":126098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110153,"num_write_kb":440486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3920,"num_bytes_recovered":15962748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126161,"ondisk_log_size":126161,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1950304,"kb_used_data":1023496,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281165216,"statfs":{"total":289910292480,"available":287913181184,"internally_reserved":0,"allocated":1048059904,"data_stored":1047312355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,5,10,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22683648,"num_objects":5538,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-180,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5538,"num_write_kb":22152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002991"},"pg_stats":[{"pgid":"2.f","version":"39'7886","reported_seq":7957,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.802525+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:16.802525+0000","last_peered":"2024-09-18T11:57:16.802525+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:16.802525+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7886,"log_dups_size":0,"ondisk_log_size":7886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32301056,"num_objects":7886,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7886,"num_write_kb":31544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4287},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'7900","reported_seq":8058,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:17.701147+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:17.701147+0000","last_peered":"2024-09-18T11:57:17.701147+0000","last_clean":"2024-09-18T11:57:17.701147+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:17.701147+0000","last_undegraded":"2024-09-18T11:57:17.701147+0000","last_fullsized":"2024-09-18T11:57:17.701147+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7900,"log_dups_size":0,"ondisk_log_size":7900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32333824,"num_objects":7894,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":20368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'7936","reported_seq":8121,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:17.551963+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:17.551963+0000","last_peered":"2024-09-18T11:57:17.551963+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:17.551963+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7936,"log_dups_size":0,"ondisk_log_size":7936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32489472,"num_objects":7932,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":19176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7827","reported_seq":7856,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:17.715811+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:17.715811+0000","last_peered":"2024-09-18T11:57:17.715811+0000","last_clean":"2024-09-18T11:57:17.715811+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:17.715811+0000","last_undegraded":"2024-09-18T11:57:17.715811+0000","last_fullsized":"2024-09-18T11:57:17.715811+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7827,"log_dups_size":0,"ondisk_log_size":7827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32059392,"num_objects":7827,"num_object_clones":0,"num_object_copies":15654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7827,"num_write_kb":31308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'7894","reported_seq":8085,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:16.904756+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:16.904756+0000","last_peered":"2024-09-18T11:57:16.904756+0000","last_clean":"2024-09-18T11:57:16.904756+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:16.904756+0000","last_undegraded":"2024-09-18T11:57:16.904756+0000","last_fullsized":"2024-09-18T11:57:16.904756+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7894,"log_dups_size":0,"ondisk_log_size":7894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32305152,"num_objects":7887,"num_object_clones":0,"num_object_copies":15774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5059,"num_write_kb":20236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'7835","reported_seq":7916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.921612+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:16.921612+0000","last_peered":"2024-09-18T11:57:16.921612+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:16.921612+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7835,"log_dups_size":0,"ondisk_log_size":7835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32072914,"num_objects":7832,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7833,"num_write_kb":31323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4305},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'7864","reported_seq":7951,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.941885+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:16.941885+0000","last_peered":"2024-09-18T11:57:16.941885+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:16.941885+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7864,"log_dups_size":0,"ondisk_log_size":7864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32187602,"num_objects":7860,"num_object_clones":0,"num_object_copies":15720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7862,"num_write_kb":31436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4301},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'7974","reported_seq":8199,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:16.975023+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:16.975023+0000","last_peered":"2024-09-18T11:57:16.975023+0000","last_clean":"2024-09-18T11:57:16.975023+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:16.975023+0000","last_undegraded":"2024-09-18T11:57:16.975023+0000","last_fullsized":"2024-09-18T11:57:16.975023+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7974,"log_dups_size":0,"ondisk_log_size":7974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32653312,"num_objects":7972,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4291,"num_write_kb":17164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7853","reported_seq":14981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:17.093103+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:17.093103+0000","last_peered":"2024-09-18T11:57:17.093103+0000","last_clean":"2024-09-18T11:57:17.093103+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:17.093103+0000","last_undegraded":"2024-09-18T11:57:17.093103+0000","last_fullsized":"2024-09-18T11:57:17.093103+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7853,"log_dups_size":0,"ondisk_log_size":7853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32138450,"num_objects":7848,"num_object_clones":0,"num_object_copies":15696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7851,"num_write_kb":31388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'7874","reported_seq":7952,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.985385+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:16.985385+0000","last_peered":"2024-09-18T11:57:16.985385+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:16.985385+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7874,"log_dups_size":0,"ondisk_log_size":7874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32220370,"num_objects":7868,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7872,"num_write_kb":31468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4336},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'7932","reported_seq":7988,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:17.666789+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:17.666789+0000","last_peered":"2024-09-18T11:57:17.666789+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:17.666789+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32470226,"num_objects":7929,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":31711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4379}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'7860","reported_seq":8069,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:17.688411+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:17.688411+0000","last_peered":"2024-09-18T11:57:17.688411+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:17.688411+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7860,"log_dups_size":0,"ondisk_log_size":7860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32169984,"num_objects":7854,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4340,"num_write_kb":17360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'7862","reported_seq":7944,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.896150+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:16.896150+0000","last_peered":"2024-09-18T11:57:16.896150+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:16.896150+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32195177,"num_objects":7861,"num_object_clones":0,"num_object_copies":15722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":31441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4326},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'7944","reported_seq":8860,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:16.976121+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:16.976121+0000","last_peered":"2024-09-18T11:57:16.976121+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:16.976121+0000","last_undegraded":"2024-09-18T11:57:16.976121+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7944,"log_dups_size":0,"ondisk_log_size":7944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32507707,"num_objects":7939,"num_object_clones":0,"num_object_copies":15878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3106,"num_objects_unfound":0,"num_objects_dirty":7939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":31748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":419,"num_bytes_recovered":1703936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4833},{"shards":"0,2","objects":3106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'7805","reported_seq":7886,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.789045+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:16.789045+0000","last_peered":"2024-09-18T11:57:16.789045+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:16.789045+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7805,"log_dups_size":0,"ondisk_log_size":7805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31954130,"num_objects":7803,"num_object_clones":0,"num_object_copies":15606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7803,"num_write_kb":31206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4210},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'7912","reported_seq":7997,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.938252+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:16.938252+0000","last_peered":"2024-09-18T11:57:16.938252+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:16.938252+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7912,"log_dups_size":0,"ondisk_log_size":7912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32360868,"num_objects":7904,"num_object_clones":0,"num_object_copies":15808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7909,"num_write_kb":31607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4381},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":516419636,"num_objects":126096,"num_object_clones":0,"num_object_copies":252192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3106,"num_objects_unfound":0,"num_objects_dirty":126096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110151,"num_write_kb":440484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3919,"num_bytes_recovered":15962729,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1049083904,"data_stored":1049003887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126158,"ondisk_log_size":126158,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051625,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672352,"kb_used_data":366168,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93699488,"statfs":{"total":96636764160,"available":95948275712,"internally_reserved":0,"allocated":374956032,"data_stored":374732457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":4.069},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":4.2229999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":4.2640000000000002},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.246}]}]},{"osd":1,"up_from":35,"seq":150323855388,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533488,"kb_used_data":223144,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93838352,"statfs":{"total":96636764160,"available":96090472448,"internally_reserved":0,"allocated":228499456,"data_stored":228202679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.252},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.31}]}]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744464,"kb_used_data":434184,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93627376,"statfs":{"total":96636764160,"available":95874433024,"internally_reserved":0,"allocated":444604416,"data_stored":444377219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":2.7130000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":2.238}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":2.379},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":2.7490000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":445001728,"data_stored":444998249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228884480,"data_stored":228811421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375197696,"data_stored":375194217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:19.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:57:19.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'7886", 'reported_seq': 7957, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:16.802525+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:57:16.802525+0000', 'last_peered': '2024-09-18T11:57:16.802525+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:57:16.802525+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7886, 'log_dups_size': 0, 'ondisk_log_size': 7886, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32301056, 'num_objects': 7886, 'num_object_clones': 0, 'num_object_copies': 15772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7886, 'num_write_kb': 31544, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4287}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:19.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:57:19.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'7936", 'reported_seq': 8121, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:17.551963+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:57:17.551963+0000', 'last_peered': '2024-09-18T11:57:17.551963+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:57:17.551963+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 7936, 'log_dups_size': 0, 'ondisk_log_size': 7936, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32489472, 'num_objects': 7932, 'num_object_clones': 0, 'num_object_copies': 15864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4794, 'num_write_kb': 19176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 4396}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:19.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:57:19.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'7835", 'reported_seq': 7916, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:16.921612+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:57:16.921612+0000', 'last_peered': '2024-09-18T11:57:16.921612+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:57:16.921612+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7835, 'log_dups_size': 0, 'ondisk_log_size': 7835, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32072914, 'num_objects': 7832, 'num_object_clones': 0, 'num_object_copies': 15664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7833, 'num_write_kb': 31323, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4305}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:19.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:57:19.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'7864", 'reported_seq': 7951, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:16.941885+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:57:16.941885+0000', 'last_peered': '2024-09-18T11:57:16.941885+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:57:16.941885+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7864, 'log_dups_size': 0, 'ondisk_log_size': 7864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32187602, 'num_objects': 7860, 'num_object_clones': 0, 'num_object_copies': 15720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7862, 'num_write_kb': 31436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4301}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:19.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:57:19.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'7874", 'reported_seq': 7952, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:16.985385+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:57:16.985385+0000', 'last_peered': '2024-09-18T11:57:16.985385+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:57:16.985385+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7874, 'log_dups_size': 0, 'ondisk_log_size': 7874, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32220370, 'num_objects': 7868, 'num_object_clones': 0, 'num_object_copies': 15736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7872, 'num_write_kb': 31468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4336}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:19.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:57:19.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'7932", 'reported_seq': 7988, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:17.666789+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:57:17.666789+0000', 'last_peered': '2024-09-18T11:57:17.666789+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:57:17.666789+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7932, 'log_dups_size': 0, 'ondisk_log_size': 7932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32470226, 'num_objects': 7929, 'num_object_clones': 0, 'num_object_copies': 15858, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7930, 'num_write_kb': 31711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 4379}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:19.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:57:19.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'7860", 'reported_seq': 8069, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:17.688411+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:57:17.688411+0000', 'last_peered': '2024-09-18T11:57:17.688411+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:57:17.688411+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 7860, 'log_dups_size': 0, 'ondisk_log_size': 7860, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32169984, 'num_objects': 7854, 'num_object_clones': 0, 'num_object_copies': 15708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4340, 'num_write_kb': 17360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 4261}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:19.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:57:19.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'7862", 'reported_seq': 7944, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:16.896150+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:57:16.896150+0000', 'last_peered': '2024-09-18T11:57:16.896150+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:57:16.896150+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7862, 'log_dups_size': 0, 'ondisk_log_size': 7862, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32195177, 'num_objects': 7861, 'num_object_clones': 0, 'num_object_copies': 15722, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7861, 'num_write_kb': 31441, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4326}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:19.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:57:19.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'7944", 'reported_seq': 8860, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:57:16.976121+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:57:16.976121+0000', 'last_peered': '2024-09-18T11:57:16.976121+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:57:16.976121+0000', 'last_undegraded': '2024-09-18T11:57:16.976121+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7944, 'log_dups_size': 0, 'ondisk_log_size': 7944, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32507707, 'num_objects': 7939, 'num_object_clones': 0, 'num_object_copies': 15878, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3106, 'num_objects_unfound': 0, 'num_objects_dirty': 7939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7941, 'num_write_kb': 31748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 419, 'num_bytes_recovered': 1703936, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4833}, {'shards': '0,2', 'objects': 3106}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:19.784 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:57:19.784 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'7805", 'reported_seq': 7886, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:16.789045+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:57:16.789045+0000', 'last_peered': '2024-09-18T11:57:16.789045+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:57:16.789045+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7805, 'log_dups_size': 0, 'ondisk_log_size': 7805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31954130, 'num_objects': 7803, 'num_object_clones': 0, 'num_object_copies': 15606, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7803, 'num_write_kb': 31206, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4210}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:19.784 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:57:19.784 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'7912", 'reported_seq': 7997, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:16.938252+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:57:16.938252+0000', 'last_peered': '2024-09-18T11:57:16.938252+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:57:16.938252+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7912, 'log_dups_size': 0, 'ondisk_log_size': 7912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32360868, 'num_objects': 7904, 'num_object_clones': 0, 'num_object_copies': 15808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7909, 'num_write_kb': 31607, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4381}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:19.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:20.144 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 174 20 127395 127375 2.85865 1.71875 0.0438042 0.0273207 2024-09-18T11:57:20.507 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:20.507 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/252196 objects degraded (14.102%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":240,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":126098,"data_bytes":516419655,"bytes_used":1997111296,"bytes_avail":287913181184,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":252196,"degraded_ratio":0.14101730400164952,"misplaced_objects":3106,"misplaced_total":252196,"misplaced_ratio":0.012315817855953306,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":61424,"recovering_keys_per_sec":0,"num_objects_recovered":180,"num_bytes_recovered":737280,"num_keys_recovered":0,"write_bytes_sec":1889832,"read_op_per_sec":0,"write_op_per_sec":461},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:20.549 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:20.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:21.145 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 175 20 127821 127801 2.85183 1.66406 0.0299851 0.0273843 2024-09-18T11:57:21.177 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:21.198 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:21.213 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-18T11:57:21.085541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516419655,"num_objects":126098,"num_object_clones":0,"num_object_copies":252196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3106,"num_objects_unfound":0,"num_objects_dirty":126098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110153,"num_write_kb":440486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3920,"num_bytes_recovered":15962748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126161,"ondisk_log_size":126161,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1950304,"kb_used_data":1023496,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281165216,"statfs":{"total":289910292480,"available":287913181184,"internally_reserved":0,"allocated":1048059904,"data_stored":1047312355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,5,10,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18927616,"num_objects":4621,"num_object_clones":0,"num_object_copies":9242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-180,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":18484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002908"},"pg_stats":[{"pgid":"2.f","version":"39'7886","reported_seq":7957,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.802525+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:16.802525+0000","last_peered":"2024-09-18T11:57:16.802525+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:16.802525+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7886,"log_dups_size":0,"ondisk_log_size":7886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32301056,"num_objects":7886,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7886,"num_write_kb":31544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4287},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'7900","reported_seq":8058,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:17.701147+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:17.701147+0000","last_peered":"2024-09-18T11:57:17.701147+0000","last_clean":"2024-09-18T11:57:17.701147+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:17.701147+0000","last_undegraded":"2024-09-18T11:57:17.701147+0000","last_fullsized":"2024-09-18T11:57:17.701147+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":7900,"log_dups_size":0,"ondisk_log_size":7900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32333824,"num_objects":7894,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":20368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'7936","reported_seq":8121,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:17.551963+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:17.551963+0000","last_peered":"2024-09-18T11:57:17.551963+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:17.551963+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":7936,"log_dups_size":0,"ondisk_log_size":7936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32489472,"num_objects":7932,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":19176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7827","reported_seq":7856,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:17.715811+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:17.715811+0000","last_peered":"2024-09-18T11:57:17.715811+0000","last_clean":"2024-09-18T11:57:17.715811+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:17.715811+0000","last_undegraded":"2024-09-18T11:57:17.715811+0000","last_fullsized":"2024-09-18T11:57:17.715811+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7827,"log_dups_size":0,"ondisk_log_size":7827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32059392,"num_objects":7827,"num_object_clones":0,"num_object_copies":15654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7827,"num_write_kb":31308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'7894","reported_seq":8085,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:16.904756+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:16.904756+0000","last_peered":"2024-09-18T11:57:16.904756+0000","last_clean":"2024-09-18T11:57:16.904756+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:16.904756+0000","last_undegraded":"2024-09-18T11:57:16.904756+0000","last_fullsized":"2024-09-18T11:57:16.904756+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":7894,"log_dups_size":0,"ondisk_log_size":7894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32305152,"num_objects":7887,"num_object_clones":0,"num_object_copies":15774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5059,"num_write_kb":20236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'7835","reported_seq":7916,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.921612+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:16.921612+0000","last_peered":"2024-09-18T11:57:16.921612+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:16.921612+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7835,"log_dups_size":0,"ondisk_log_size":7835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32072914,"num_objects":7832,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7833,"num_write_kb":31323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4305},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'7864","reported_seq":7951,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.941885+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:16.941885+0000","last_peered":"2024-09-18T11:57:16.941885+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:16.941885+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7864,"log_dups_size":0,"ondisk_log_size":7864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32187602,"num_objects":7860,"num_object_clones":0,"num_object_copies":15720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7862,"num_write_kb":31436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4301},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'7974","reported_seq":8199,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:16.975023+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:16.975023+0000","last_peered":"2024-09-18T11:57:16.975023+0000","last_clean":"2024-09-18T11:57:16.975023+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:16.975023+0000","last_undegraded":"2024-09-18T11:57:16.975023+0000","last_fullsized":"2024-09-18T11:57:16.975023+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":7974,"log_dups_size":0,"ondisk_log_size":7974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32653312,"num_objects":7972,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4291,"num_write_kb":17164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7853","reported_seq":14981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:17.093103+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:17.093103+0000","last_peered":"2024-09-18T11:57:17.093103+0000","last_clean":"2024-09-18T11:57:17.093103+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:17.093103+0000","last_undegraded":"2024-09-18T11:57:17.093103+0000","last_fullsized":"2024-09-18T11:57:17.093103+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7853,"log_dups_size":0,"ondisk_log_size":7853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32138450,"num_objects":7848,"num_object_clones":0,"num_object_copies":15696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7851,"num_write_kb":31388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'7874","reported_seq":7952,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.985385+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:16.985385+0000","last_peered":"2024-09-18T11:57:16.985385+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:16.985385+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7874,"log_dups_size":0,"ondisk_log_size":7874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32220370,"num_objects":7868,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7872,"num_write_kb":31468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4336},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'7932","reported_seq":7988,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:17.666789+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:17.666789+0000","last_peered":"2024-09-18T11:57:17.666789+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:17.666789+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32470226,"num_objects":7929,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7930,"num_write_kb":31711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4379}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'7860","reported_seq":8069,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:17.688411+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:17.688411+0000","last_peered":"2024-09-18T11:57:17.688411+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:17.688411+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":7860,"log_dups_size":0,"ondisk_log_size":7860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32169984,"num_objects":7854,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4340,"num_write_kb":17360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'7862","reported_seq":7944,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.896150+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:16.896150+0000","last_peered":"2024-09-18T11:57:16.896150+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:16.896150+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32195177,"num_objects":7861,"num_object_clones":0,"num_object_copies":15722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":31441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4326},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'7944","reported_seq":8860,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:16.976121+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:16.976121+0000","last_peered":"2024-09-18T11:57:16.976121+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:16.976121+0000","last_undegraded":"2024-09-18T11:57:16.976121+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7944,"log_dups_size":0,"ondisk_log_size":7944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32507707,"num_objects":7939,"num_object_clones":0,"num_object_copies":15878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3106,"num_objects_unfound":0,"num_objects_dirty":7939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":31748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":419,"num_bytes_recovered":1703936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4833},{"shards":"0,2","objects":3106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'7805","reported_seq":7886,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.789045+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:16.789045+0000","last_peered":"2024-09-18T11:57:16.789045+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:16.789045+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7805,"log_dups_size":0,"ondisk_log_size":7805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31954130,"num_objects":7803,"num_object_clones":0,"num_object_copies":15606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7803,"num_write_kb":31206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4210},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'7912","reported_seq":7997,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:16.938252+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:16.938252+0000","last_peered":"2024-09-18T11:57:16.938252+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:16.938252+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7912,"log_dups_size":0,"ondisk_log_size":7912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32360868,"num_objects":7904,"num_object_clones":0,"num_object_copies":15808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7909,"num_write_kb":31607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4381},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":516419636,"num_objects":126096,"num_object_clones":0,"num_object_copies":252192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":3106,"num_objects_unfound":0,"num_objects_dirty":126096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110151,"num_write_kb":440484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3919,"num_bytes_recovered":15962729,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1049083904,"data_stored":1049003887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126158,"ondisk_log_size":126158,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051625,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672352,"kb_used_data":366168,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93699488,"statfs":{"total":96636764160,"available":95948275712,"internally_reserved":0,"allocated":374956032,"data_stored":374732457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.44800000000000001,"15min":0.40100000000000002},"min":{"1min":0.191,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":4.069},{"interface":"front","average":{"1min":1.804,"5min":0.59399999999999997,"15min":0.39200000000000002},"min":{"1min":0.214,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":4.2229999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":4.2640000000000002},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.246}]}]},{"osd":1,"up_from":35,"seq":150323855388,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533488,"kb_used_data":223144,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93838352,"statfs":{"total":96636764160,"available":96090472448,"internally_reserved":0,"allocated":228499456,"data_stored":228202679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.252},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.31}]}]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744464,"kb_used_data":434184,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93627376,"statfs":{"total":96636764160,"available":95874433024,"internally_reserved":0,"allocated":444604416,"data_stored":444377219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":2.7130000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":2.238}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":2.379},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":2.7490000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":445001728,"data_stored":444998249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228884480,"data_stored":228811421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375197696,"data_stored":375194217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:22.145 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 176 20 128283 128263 2.84587 1.80469 0.022303 0.0274447 2024-09-18T11:57:23.145 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 177 20 128754 128734 2.84019 1.83984 0.0649786 0.0274981 2024-09-18T11:57:24.148 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 178 20 129226 129206 2.83455 1.84375 0.0349833 0.0275528 2024-09-18T11:57:24.215 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:24.870 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:24.882 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:24.894 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-18T11:57:23.085970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":525639751,"num_objects":128349,"num_object_clones":0,"num_object_copies":256698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2998,"num_objects_unfound":0,"num_objects_dirty":128349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112404,"num_write_kb":449490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4028,"num_bytes_recovered":16405116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128412,"ondisk_log_size":128412,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1947672,"kb_used_data":1041664,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281167848,"statfs":{"total":289910292480,"available":287915876352,"internally_reserved":0,"allocated":1066663936,"data_stored":1065916387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,2,4,9,8,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28147712,"num_objects":6872,"num_object_clones":0,"num_object_copies":13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-288,"num_objects_unfound":0,"num_objects_dirty":6872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":27488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002833"},"pg_stats":[{"pgid":"2.f","version":"39'8028","reported_seq":8099,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.983154+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:21.983154+0000","last_peered":"2024-09-18T11:57:21.983154+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:21.983154+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8028,"log_dups_size":0,"ondisk_log_size":8028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32882688,"num_objects":8028,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8028,"num_write_kb":32112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4429},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8047","reported_seq":8205,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.690754+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:22.690754+0000","last_peered":"2024-09-18T11:57:22.690754+0000","last_clean":"2024-09-18T11:57:22.690754+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:22.690754+0000","last_undegraded":"2024-09-18T11:57:22.690754+0000","last_fullsized":"2024-09-18T11:57:22.690754+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32935936,"num_objects":8041,"num_object_clones":0,"num_object_copies":16082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5239,"num_write_kb":20956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8082","reported_seq":8265,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:22.671471+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:22.671471+0000","last_peered":"2024-09-18T11:57:22.671471+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:22.671471+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33087488,"num_objects":8078,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4940,"num_write_kb":19760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4542}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7964","reported_seq":7993,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.748362+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:22.748362+0000","last_peered":"2024-09-18T11:57:22.748362+0000","last_clean":"2024-09-18T11:57:22.748362+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:22.748362+0000","last_undegraded":"2024-09-18T11:57:22.748362+0000","last_fullsized":"2024-09-18T11:57:22.748362+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7964,"log_dups_size":0,"ondisk_log_size":7964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32620544,"num_objects":7964,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7964,"num_write_kb":31856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8043","reported_seq":8234,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.998313+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:21.998313+0000","last_peered":"2024-09-18T11:57:21.998313+0000","last_clean":"2024-09-18T11:57:21.998313+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:21.998313+0000","last_undegraded":"2024-09-18T11:57:21.998313+0000","last_fullsized":"2024-09-18T11:57:21.998313+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32915456,"num_objects":8036,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5208,"num_write_kb":20832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'7997","reported_seq":8078,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.939864+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:21.939864+0000","last_peered":"2024-09-18T11:57:21.939864+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:21.939864+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7997,"log_dups_size":0,"ondisk_log_size":7997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32736466,"num_objects":7994,"num_object_clones":0,"num_object_copies":15988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7995,"num_write_kb":31971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4467},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'7992","reported_seq":8077,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.973317+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:21.973317+0000","last_peered":"2024-09-18T11:57:21.973317+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:21.973317+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7992,"log_dups_size":0,"ondisk_log_size":7992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32711890,"num_objects":7988,"num_object_clones":0,"num_object_copies":15976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7990,"num_write_kb":31948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4429},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8105","reported_seq":8330,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.980937+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:21.980937+0000","last_peered":"2024-09-18T11:57:21.980937+0000","last_clean":"2024-09-18T11:57:21.980937+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:21.980937+0000","last_undegraded":"2024-09-18T11:57:21.980937+0000","last_fullsized":"2024-09-18T11:57:21.980937+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8105,"log_dups_size":0,"ondisk_log_size":8105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33189888,"num_objects":8103,"num_object_clones":0,"num_object_copies":16206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":17688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7984","reported_seq":15112,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.192420+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:22.192420+0000","last_peered":"2024-09-18T11:57:22.192420+0000","last_clean":"2024-09-18T11:57:22.192420+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:22.192420+0000","last_undegraded":"2024-09-18T11:57:22.192420+0000","last_fullsized":"2024-09-18T11:57:22.192420+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7984,"log_dups_size":0,"ondisk_log_size":7984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32675026,"num_objects":7979,"num_object_clones":0,"num_object_copies":15958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7982,"num_write_kb":31912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8015","reported_seq":8093,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.955959+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:21.955959+0000","last_peered":"2024-09-18T11:57:21.955959+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:21.955959+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8015,"log_dups_size":0,"ondisk_log_size":8015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32797906,"num_objects":8009,"num_object_clones":0,"num_object_copies":16018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8013,"num_write_kb":32032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4477},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8072","reported_seq":8127,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:22.749093+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:22.749093+0000","last_peered":"2024-09-18T11:57:22.749093+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:22.749093+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33043666,"num_objects":8069,"num_object_clones":0,"num_object_copies":16138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":32271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4519}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8017","reported_seq":8226,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:22.750859+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:22.750859+0000","last_peered":"2024-09-18T11:57:22.750859+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:22.750859+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8017,"log_dups_size":0,"ondisk_log_size":8017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32813056,"num_objects":8011,"num_object_clones":0,"num_object_copies":16022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4497,"num_write_kb":17988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4418}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8002","reported_seq":8084,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.924215+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:21.924215+0000","last_peered":"2024-09-18T11:57:21.924215+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:21.924215+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8002,"log_dups_size":0,"ondisk_log_size":8002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768617,"num_objects":8001,"num_object_clones":0,"num_object_copies":16002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8001,"num_write_kb":32001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4466},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8070","reported_seq":9203,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:21.988806+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:21.988806+0000","last_peered":"2024-09-18T11:57:21.988806+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:21.988806+0000","last_undegraded":"2024-09-18T11:57:21.988806+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8070,"log_dups_size":0,"ondisk_log_size":8070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33023803,"num_objects":8065,"num_object_clones":0,"num_object_copies":16130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2998,"num_objects_unfound":0,"num_objects_dirty":8065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8067,"num_write_kb":32252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":527,"num_bytes_recovered":2146304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5067},{"shards":"0,2","objects":2998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'7926","reported_seq":8007,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.965728+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:21.965728+0000","last_peered":"2024-09-18T11:57:21.965728+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:21.965728+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7926,"log_dups_size":0,"ondisk_log_size":7926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32449746,"num_objects":7924,"num_object_clones":0,"num_object_copies":15848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7924,"num_write_kb":31690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4331},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8065","reported_seq":8150,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.958403+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:21.958403+0000","last_peered":"2024-09-18T11:57:21.958403+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:21.958403+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8065,"log_dups_size":0,"ondisk_log_size":8065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32987556,"num_objects":8057,"num_object_clones":0,"num_object_copies":16114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8062,"num_write_kb":32219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4534},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":525639732,"num_objects":128347,"num_object_clones":0,"num_object_copies":256694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2998,"num_objects_unfound":0,"num_objects_dirty":128347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112402,"num_write_kb":449488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4027,"num_bytes_recovered":16405097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1067835392,"data_stored":1067755375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128409,"ondisk_log_size":128409,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051626,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680944,"kb_used_data":370600,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93690896,"statfs":{"total":96636764160,"available":95939477504,"internally_reserved":0,"allocated":379494400,"data_stored":379270825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.24099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.27900000000000003},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.29099999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855389,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528072,"kb_used_data":230208,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93843768,"statfs":{"total":96636764160,"available":96096018432,"internally_reserved":0,"allocated":235732992,"data_stored":235436215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.19},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":4.8799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.221}]}]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738656,"kb_used_data":440856,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93633184,"statfs":{"total":96636764160,"available":95880380416,"internally_reserved":0,"allocated":451436544,"data_stored":451209347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,3,3,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.186},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.249}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":8.6080000000000005},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":9.4309999999999992}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":451846144,"data_stored":451842665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236179456,"data_stored":236106397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":379809792,"data_stored":379806313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:24.896 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:25.148 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 179 20 129649 129629 2.82795 1.65234 0.0244088 0.0276181 2024-09-18T11:57:25.737 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:25.745 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:25.760 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-18T11:57:25.086487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":525639751,"num_objects":128349,"num_object_clones":0,"num_object_copies":256698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2998,"num_objects_unfound":0,"num_objects_dirty":128349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112404,"num_write_kb":449490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4028,"num_bytes_recovered":16405116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128412,"ondisk_log_size":128412,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1947672,"kb_used_data":1041664,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281167848,"statfs":{"total":289910292480,"available":287915876352,"internally_reserved":0,"allocated":1066663936,"data_stored":1065916387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,2,4,9,8,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18657280,"num_objects":4555,"num_object_clones":0,"num_object_copies":9110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-195,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4555,"num_write_kb":18220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":798720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002861"},"pg_stats":[{"pgid":"2.f","version":"39'8028","reported_seq":8099,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.983154+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:21.983154+0000","last_peered":"2024-09-18T11:57:21.983154+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:21.983154+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8028,"log_dups_size":0,"ondisk_log_size":8028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32882688,"num_objects":8028,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8028,"num_write_kb":32112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4429},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8047","reported_seq":8205,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.690754+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:22.690754+0000","last_peered":"2024-09-18T11:57:22.690754+0000","last_clean":"2024-09-18T11:57:22.690754+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:22.690754+0000","last_undegraded":"2024-09-18T11:57:22.690754+0000","last_fullsized":"2024-09-18T11:57:22.690754+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32935936,"num_objects":8041,"num_object_clones":0,"num_object_copies":16082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5239,"num_write_kb":20956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8082","reported_seq":8265,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:22.671471+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:22.671471+0000","last_peered":"2024-09-18T11:57:22.671471+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:22.671471+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33087488,"num_objects":8078,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4940,"num_write_kb":19760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4542}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7964","reported_seq":7993,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.748362+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:22.748362+0000","last_peered":"2024-09-18T11:57:22.748362+0000","last_clean":"2024-09-18T11:57:22.748362+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:22.748362+0000","last_undegraded":"2024-09-18T11:57:22.748362+0000","last_fullsized":"2024-09-18T11:57:22.748362+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7964,"log_dups_size":0,"ondisk_log_size":7964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32620544,"num_objects":7964,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7964,"num_write_kb":31856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8043","reported_seq":8234,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.998313+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:21.998313+0000","last_peered":"2024-09-18T11:57:21.998313+0000","last_clean":"2024-09-18T11:57:21.998313+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:21.998313+0000","last_undegraded":"2024-09-18T11:57:21.998313+0000","last_fullsized":"2024-09-18T11:57:21.998313+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32915456,"num_objects":8036,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5208,"num_write_kb":20832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'7997","reported_seq":8078,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.939864+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:21.939864+0000","last_peered":"2024-09-18T11:57:21.939864+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:21.939864+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7997,"log_dups_size":0,"ondisk_log_size":7997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32736466,"num_objects":7994,"num_object_clones":0,"num_object_copies":15988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7995,"num_write_kb":31971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4467},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'7992","reported_seq":8077,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.973317+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:21.973317+0000","last_peered":"2024-09-18T11:57:21.973317+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:21.973317+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7992,"log_dups_size":0,"ondisk_log_size":7992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32711890,"num_objects":7988,"num_object_clones":0,"num_object_copies":15976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7990,"num_write_kb":31948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4429},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8105","reported_seq":8330,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:21.980937+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:21.980937+0000","last_peered":"2024-09-18T11:57:21.980937+0000","last_clean":"2024-09-18T11:57:21.980937+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:21.980937+0000","last_undegraded":"2024-09-18T11:57:21.980937+0000","last_fullsized":"2024-09-18T11:57:21.980937+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8105,"log_dups_size":0,"ondisk_log_size":8105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33189888,"num_objects":8103,"num_object_clones":0,"num_object_copies":16206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":17688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7984","reported_seq":15112,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.192420+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:22.192420+0000","last_peered":"2024-09-18T11:57:22.192420+0000","last_clean":"2024-09-18T11:57:22.192420+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:22.192420+0000","last_undegraded":"2024-09-18T11:57:22.192420+0000","last_fullsized":"2024-09-18T11:57:22.192420+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7984,"log_dups_size":0,"ondisk_log_size":7984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32675026,"num_objects":7979,"num_object_clones":0,"num_object_copies":15958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7982,"num_write_kb":31912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8015","reported_seq":8093,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.955959+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:21.955959+0000","last_peered":"2024-09-18T11:57:21.955959+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:21.955959+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8015,"log_dups_size":0,"ondisk_log_size":8015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32797906,"num_objects":8009,"num_object_clones":0,"num_object_copies":16018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8013,"num_write_kb":32032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4477},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8072","reported_seq":8127,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:22.749093+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:22.749093+0000","last_peered":"2024-09-18T11:57:22.749093+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:22.749093+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33043666,"num_objects":8069,"num_object_clones":0,"num_object_copies":16138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":32271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4519}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8017","reported_seq":8226,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:22.750859+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:22.750859+0000","last_peered":"2024-09-18T11:57:22.750859+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:22.750859+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8017,"log_dups_size":0,"ondisk_log_size":8017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32813056,"num_objects":8011,"num_object_clones":0,"num_object_copies":16022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4497,"num_write_kb":17988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4418}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8002","reported_seq":8084,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.924215+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:21.924215+0000","last_peered":"2024-09-18T11:57:21.924215+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:21.924215+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8002,"log_dups_size":0,"ondisk_log_size":8002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768617,"num_objects":8001,"num_object_clones":0,"num_object_copies":16002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8001,"num_write_kb":32001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4466},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8070","reported_seq":9203,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:21.988806+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:21.988806+0000","last_peered":"2024-09-18T11:57:21.988806+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:21.988806+0000","last_undegraded":"2024-09-18T11:57:21.988806+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8070,"log_dups_size":0,"ondisk_log_size":8070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33023803,"num_objects":8065,"num_object_clones":0,"num_object_copies":16130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2998,"num_objects_unfound":0,"num_objects_dirty":8065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8067,"num_write_kb":32252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":527,"num_bytes_recovered":2146304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5067},{"shards":"0,2","objects":2998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'7926","reported_seq":8007,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.965728+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:21.965728+0000","last_peered":"2024-09-18T11:57:21.965728+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:21.965728+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7926,"log_dups_size":0,"ondisk_log_size":7926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32449746,"num_objects":7924,"num_object_clones":0,"num_object_copies":15848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7924,"num_write_kb":31690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4331},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8065","reported_seq":8150,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:21.958403+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:21.958403+0000","last_peered":"2024-09-18T11:57:21.958403+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:21.958403+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8065,"log_dups_size":0,"ondisk_log_size":8065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32987556,"num_objects":8057,"num_object_clones":0,"num_object_copies":16114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8062,"num_write_kb":32219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4534},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":525639732,"num_objects":128347,"num_object_clones":0,"num_object_copies":256694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2998,"num_objects_unfound":0,"num_objects_dirty":128347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112402,"num_write_kb":449488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4027,"num_bytes_recovered":16405097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1067835392,"data_stored":1067755375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128409,"ondisk_log_size":128409,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051626,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680944,"kb_used_data":370600,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93690896,"statfs":{"total":96636764160,"available":95939477504,"internally_reserved":0,"allocated":379494400,"data_stored":379270825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.24099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.27900000000000003},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.29099999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855389,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528072,"kb_used_data":230208,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93843768,"statfs":{"total":96636764160,"available":96096018432,"internally_reserved":0,"allocated":235732992,"data_stored":235436215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.19},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":4.8799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.221}]}]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738656,"kb_used_data":440856,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93633184,"statfs":{"total":96636764160,"available":95880380416,"internally_reserved":0,"allocated":451436544,"data_stored":451209347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,3,3,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.186},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.249}]},{"osd":2,"last update":"Wed Sep 18 11:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.185,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.181,"5min":5.181,"15min":5.181},"last":8.6080000000000005},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.41799999999999998,"15min":0.35399999999999998},"min":{"1min":0.193,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.7140000000000004,"5min":4.7140000000000004,"15min":4.7140000000000004},"last":9.4309999999999992}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":451846144,"data_stored":451842665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236179456,"data_stored":236106397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":379809792,"data_stored":379806313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:25.761 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:57:25.761 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'8028", 'reported_seq': 8099, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.983154+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:57:21.983154+0000', 'last_peered': '2024-09-18T11:57:21.983154+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:57:21.983154+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8028, 'log_dups_size': 0, 'ondisk_log_size': 8028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32882688, 'num_objects': 8028, 'num_object_clones': 0, 'num_object_copies': 16056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8028, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8028, 'num_write_kb': 32112, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4429}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:25.762 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:57:25.762 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'8082", 'reported_seq': 8265, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:22.671471+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:57:22.671471+0000', 'last_peered': '2024-09-18T11:57:22.671471+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:57:22.671471+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 8082, 'log_dups_size': 0, 'ondisk_log_size': 8082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33087488, 'num_objects': 8078, 'num_object_clones': 0, 'num_object_copies': 16156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8078, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4940, 'num_write_kb': 19760, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 4542}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:25.762 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:57:25.762 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'7997", 'reported_seq': 8078, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.939864+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:57:21.939864+0000', 'last_peered': '2024-09-18T11:57:21.939864+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:57:21.939864+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7997, 'log_dups_size': 0, 'ondisk_log_size': 7997, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32736466, 'num_objects': 7994, 'num_object_clones': 0, 'num_object_copies': 15988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7995, 'num_write_kb': 31971, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4467}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:25.762 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:57:25.762 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'7992", 'reported_seq': 8077, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.973317+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:57:21.973317+0000', 'last_peered': '2024-09-18T11:57:21.973317+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:57:21.973317+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7992, 'log_dups_size': 0, 'ondisk_log_size': 7992, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32711890, 'num_objects': 7988, 'num_object_clones': 0, 'num_object_copies': 15976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7990, 'num_write_kb': 31948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4429}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:25.762 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:57:25.762 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'8015", 'reported_seq': 8093, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.955959+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:57:21.955959+0000', 'last_peered': '2024-09-18T11:57:21.955959+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:57:21.955959+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8015, 'log_dups_size': 0, 'ondisk_log_size': 8015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32797906, 'num_objects': 8009, 'num_object_clones': 0, 'num_object_copies': 16018, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8009, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8013, 'num_write_kb': 32032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4477}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:25.762 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:57:25.762 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'8072", 'reported_seq': 8127, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:22.749093+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:57:22.749093+0000', 'last_peered': '2024-09-18T11:57:22.749093+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:57:22.749093+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8072, 'log_dups_size': 0, 'ondisk_log_size': 8072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33043666, 'num_objects': 8069, 'num_object_clones': 0, 'num_object_copies': 16138, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8070, 'num_write_kb': 32271, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 4519}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:25.763 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:57:25.763 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'8017", 'reported_seq': 8226, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:22.750859+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:57:22.750859+0000', 'last_peered': '2024-09-18T11:57:22.750859+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:57:22.750859+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 8017, 'log_dups_size': 0, 'ondisk_log_size': 8017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32813056, 'num_objects': 8011, 'num_object_clones': 0, 'num_object_copies': 16022, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4497, 'num_write_kb': 17988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 4418}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:25.763 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:57:25.763 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'8002", 'reported_seq': 8084, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.924215+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:57:21.924215+0000', 'last_peered': '2024-09-18T11:57:21.924215+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:57:21.924215+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8002, 'log_dups_size': 0, 'ondisk_log_size': 8002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32768617, 'num_objects': 8001, 'num_object_clones': 0, 'num_object_copies': 16002, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8001, 'num_write_kb': 32001, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4466}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:25.763 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:57:25.763 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'8070", 'reported_seq': 9203, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:57:21.988806+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:57:21.988806+0000', 'last_peered': '2024-09-18T11:57:21.988806+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:57:21.988806+0000', 'last_undegraded': '2024-09-18T11:57:21.988806+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8070, 'log_dups_size': 0, 'ondisk_log_size': 8070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33023803, 'num_objects': 8065, 'num_object_clones': 0, 'num_object_copies': 16130, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2998, 'num_objects_unfound': 0, 'num_objects_dirty': 8065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8067, 'num_write_kb': 32252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 527, 'num_bytes_recovered': 2146304, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5067}, {'shards': '0,2', 'objects': 2998}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:25.763 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:57:25.763 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'7926", 'reported_seq': 8007, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.965728+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:57:21.965728+0000', 'last_peered': '2024-09-18T11:57:21.965728+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:57:21.965728+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 7926, 'log_dups_size': 0, 'ondisk_log_size': 7926, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32449746, 'num_objects': 7924, 'num_object_clones': 0, 'num_object_copies': 15848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7924, 'num_write_kb': 31690, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4331}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:25.763 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:57:25.763 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'8065", 'reported_seq': 8150, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:21.958403+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:57:21.958403+0000', 'last_peered': '2024-09-18T11:57:21.958403+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:57:21.958403+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8065, 'log_dups_size': 0, 'ondisk_log_size': 8065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32987556, 'num_objects': 8057, 'num_object_clones': 0, 'num_object_copies': 16114, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8057, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8062, 'num_write_kb': 32219, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4534}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:25.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:26.148 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:2024-09-18T11:57:26.149129+0000 min lat: 0.0021114 max lat: 2.57663 avg lat: 0.0276802 2024-09-18T11:57:26.148 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:57:26.148 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 180 20 130072 130052 2.82142 1.65234 0.120308 0.0276802 2024-09-18T11:57:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:26.625 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/256698 objects degraded (13.854%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":246,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":128349,"data_bytes":525639751,"bytes_used":1994416128,"bytes_avail":287915876352,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":256698,"degraded_ratio":0.13854412578204739,"misplaced_objects":2998,"misplaced_total":256698,"misplaced_ratio":0.011679093721026263,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":66544,"recovering_keys_per_sec":0,"num_objects_recovered":195,"num_bytes_recovered":798720,"num_keys_recovered":0,"write_bytes_sec":1554402,"read_op_per_sec":0,"write_op_per_sec":379},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:26.654 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:26.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:27.148 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 181 20 130503 130483 2.81513 1.68359 0.0641895 0.0277433 2024-09-18T11:57:27.476 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:27.505 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:27.528 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-18T11:57:27.086935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":531308615,"num_objects":129733,"num_object_clones":0,"num_object_copies":259466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2906,"num_objects_unfound":0,"num_objects_dirty":129733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113788,"num_write_kb":455026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4120,"num_bytes_recovered":16781948,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129796,"ondisk_log_size":129796,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1962596,"kb_used_data":1048268,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281152924,"statfs":{"total":289910292480,"available":287900594176,"internally_reserved":0,"allocated":1073426432,"data_stored":1072678883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,1,2,9,13,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24326144,"num_objects":5939,"num_object_clones":0,"num_object_copies":11878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-287,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5939,"num_write_kb":23756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":1175552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002811"},"pg_stats":[{"pgid":"2.f","version":"39'8165","reported_seq":8236,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.980275+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:26.980275+0000","last_peered":"2024-09-18T11:57:26.980275+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:26.980275+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8165,"log_dups_size":0,"ondisk_log_size":8165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33443840,"num_objects":8165,"num_object_clones":0,"num_object_copies":16330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8165,"num_write_kb":32660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4566},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8047","reported_seq":8205,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.690754+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:22.690754+0000","last_peered":"2024-09-18T11:57:22.690754+0000","last_clean":"2024-09-18T11:57:22.690754+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:22.690754+0000","last_undegraded":"2024-09-18T11:57:22.690754+0000","last_fullsized":"2024-09-18T11:57:22.690754+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32935936,"num_objects":8041,"num_object_clones":0,"num_object_copies":16082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5239,"num_write_kb":20956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8082","reported_seq":8265,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:22.671471+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:22.671471+0000","last_peered":"2024-09-18T11:57:22.671471+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:22.671471+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8082,"log_dups_size":0,"ondisk_log_size":8082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33087488,"num_objects":8078,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4940,"num_write_kb":19760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4542}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'7964","reported_seq":7993,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.748362+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:22.748362+0000","last_peered":"2024-09-18T11:57:22.748362+0000","last_clean":"2024-09-18T11:57:22.748362+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:22.748362+0000","last_undegraded":"2024-09-18T11:57:22.748362+0000","last_fullsized":"2024-09-18T11:57:22.748362+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":7964,"log_dups_size":0,"ondisk_log_size":7964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32620544,"num_objects":7964,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7964,"num_write_kb":31856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8181","reported_seq":8372,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:26.980838+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:26.980838+0000","last_peered":"2024-09-18T11:57:26.980838+0000","last_clean":"2024-09-18T11:57:26.980838+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:26.980838+0000","last_undegraded":"2024-09-18T11:57:26.980838+0000","last_fullsized":"2024-09-18T11:57:26.980838+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8181,"log_dups_size":0,"ondisk_log_size":8181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33480704,"num_objects":8174,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5346,"num_write_kb":21384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8139","reported_seq":8220,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.986434+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:26.986434+0000","last_peered":"2024-09-18T11:57:26.986434+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:26.986434+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8139,"log_dups_size":0,"ondisk_log_size":8139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33318098,"num_objects":8136,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8137,"num_write_kb":32539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4609},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8127","reported_seq":8214,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.896646+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:26.896646+0000","last_peered":"2024-09-18T11:57:26.896646+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:26.896646+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264850,"num_objects":8123,"num_object_clones":0,"num_object_copies":16246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":32488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4564},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8245","reported_seq":8470,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:26.924111+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:26.924111+0000","last_peered":"2024-09-18T11:57:26.924111+0000","last_clean":"2024-09-18T11:57:26.924111+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:26.924111+0000","last_undegraded":"2024-09-18T11:57:26.924111+0000","last_fullsized":"2024-09-18T11:57:26.924111+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8245,"log_dups_size":0,"ondisk_log_size":8245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33763328,"num_objects":8243,"num_object_clones":0,"num_object_copies":16486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":18248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'7984","reported_seq":15112,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:22.192420+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:22.192420+0000","last_peered":"2024-09-18T11:57:22.192420+0000","last_clean":"2024-09-18T11:57:22.192420+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:22.192420+0000","last_undegraded":"2024-09-18T11:57:22.192420+0000","last_fullsized":"2024-09-18T11:57:22.192420+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":7984,"log_dups_size":0,"ondisk_log_size":7984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32675026,"num_objects":7979,"num_object_clones":0,"num_object_copies":15958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7982,"num_write_kb":31912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8137","reported_seq":8215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.933162+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:26.933162+0000","last_peered":"2024-09-18T11:57:26.933162+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:26.933162+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8137,"log_dups_size":0,"ondisk_log_size":8137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33297618,"num_objects":8131,"num_object_clones":0,"num_object_copies":16262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8135,"num_write_kb":32520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4599},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8072","reported_seq":8127,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:22.749093+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:22.749093+0000","last_peered":"2024-09-18T11:57:22.749093+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:22.749093+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33043666,"num_objects":8069,"num_object_clones":0,"num_object_copies":16138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":32271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4519}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8017","reported_seq":8226,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:22.750859+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:22.750859+0000","last_peered":"2024-09-18T11:57:22.750859+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:22.750859+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8017,"log_dups_size":0,"ondisk_log_size":8017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32813056,"num_objects":8011,"num_object_clones":0,"num_object_copies":16022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4497,"num_write_kb":17988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4418}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8137","reported_seq":8219,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.965167+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:26.965167+0000","last_peered":"2024-09-18T11:57:26.965167+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:26.965167+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8137,"log_dups_size":0,"ondisk_log_size":8137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33321577,"num_objects":8136,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8136,"num_write_kb":32541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4601},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8217","reported_seq":9534,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:27.017753+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:27.017753+0000","last_peered":"2024-09-18T11:57:27.017753+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:27.017753+0000","last_undegraded":"2024-09-18T11:57:27.017753+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33625915,"num_objects":8212,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2906,"num_objects_unfound":0,"num_objects_dirty":8212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":32840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":619,"num_bytes_recovered":2523136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5306},{"shards":"0,2","objects":2906}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8069","reported_seq":8147,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.998338+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:26.998338+0000","last_peered":"2024-09-18T11:57:26.998338+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:26.998338+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8069,"log_dups_size":0,"ondisk_log_size":8069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33035474,"num_objects":8067,"num_object_clones":0,"num_object_copies":16134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8067,"num_write_kb":32262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4474},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8210","reported_seq":8295,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:27.005100+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:27.005100+0000","last_peered":"2024-09-18T11:57:27.005100+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:27.005100+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8210,"log_dups_size":0,"ondisk_log_size":8210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33581476,"num_objects":8202,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":32799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4679},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":531308596,"num_objects":129731,"num_object_clones":0,"num_object_copies":259462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2906,"num_objects_unfound":0,"num_objects_dirty":129731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113786,"num_write_kb":455024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4119,"num_bytes_recovered":16781929,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1074692096,"data_stored":1074612079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129793,"ondisk_log_size":129793,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051626,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680944,"kb_used_data":370600,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93690896,"statfs":{"total":96636764160,"available":95939477504,"internally_reserved":0,"allocated":379494400,"data_stored":379270825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.24099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.27900000000000003},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.29099999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855389,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528072,"kb_used_data":230208,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93843768,"statfs":{"total":96636764160,"available":96096018432,"internally_reserved":0,"allocated":235732992,"data_stored":235436215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.19},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":4.8799999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.221}]}]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753580,"kb_used_data":447460,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93618260,"statfs":{"total":96636764160,"available":95865098240,"internally_reserved":0,"allocated":458199040,"data_stored":457971843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.31900000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.38300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":458702848,"data_stored":458699369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236179456,"data_stored":236106397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":379809792,"data_stored":379806313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:28.148 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 182 20 130966 130946 2.8096 1.80859 0.0713344 0.0277981 2024-09-18T11:57:29.148 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 183 20 131418 131398 2.8039 1.76562 0.0801627 0.027855 2024-09-18T11:57:30.148 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 184 20 131876 131856 2.79838 1.78906 0.037341 0.0279077 2024-09-18T11:57:30.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:31.148 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 185 20 132326 132306 2.79276 1.75781 0.0297756 0.0279654 2024-09-18T11:57:31.362 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:31.402 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:31.410 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-18T11:57:31.087888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534741063,"num_objects":130571,"num_object_clones":0,"num_object_copies":261142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2906,"num_objects_unfound":0,"num_objects_dirty":130571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114626,"num_write_kb":458378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4120,"num_bytes_recovered":16781948,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130634,"ondisk_log_size":130634,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1969972,"kb_used_data":1059804,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281145548,"statfs":{"total":289910292480,"available":287893041152,"internally_reserved":0,"allocated":1085239296,"data_stored":1084491747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,1,2,5,9,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18321408,"num_objects":4473,"num_object_clones":0,"num_object_copies":8946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-200,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4473,"num_write_kb":17892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002818"},"pg_stats":[{"pgid":"2.f","version":"39'8165","reported_seq":8236,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.980275+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:26.980275+0000","last_peered":"2024-09-18T11:57:26.980275+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:26.980275+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8165,"log_dups_size":0,"ondisk_log_size":8165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33443840,"num_objects":8165,"num_object_clones":0,"num_object_copies":16330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8165,"num_write_kb":32660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4566},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8181","reported_seq":8339,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:27.707941+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:27.707941+0000","last_peered":"2024-09-18T11:57:27.707941+0000","last_clean":"2024-09-18T11:57:27.707941+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:27.707941+0000","last_undegraded":"2024-09-18T11:57:27.707941+0000","last_fullsized":"2024-09-18T11:57:27.707941+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8181,"log_dups_size":0,"ondisk_log_size":8181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33484800,"num_objects":8175,"num_object_clones":0,"num_object_copies":16350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5373,"num_write_kb":21492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8228","reported_seq":8413,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:27.723770+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:27.723770+0000","last_peered":"2024-09-18T11:57:27.723770+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:27.723770+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8228,"log_dups_size":0,"ondisk_log_size":8228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":8224,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5086,"num_write_kb":20344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4688}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8115","reported_seq":8144,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:27.745137+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:27.745137+0000","last_peered":"2024-09-18T11:57:27.745137+0000","last_clean":"2024-09-18T11:57:27.745137+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:27.745137+0000","last_undegraded":"2024-09-18T11:57:27.745137+0000","last_fullsized":"2024-09-18T11:57:27.745137+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8115,"log_dups_size":0,"ondisk_log_size":8115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33239040,"num_objects":8115,"num_object_clones":0,"num_object_copies":16230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8115,"num_write_kb":32460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8181","reported_seq":8372,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:26.980838+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:26.980838+0000","last_peered":"2024-09-18T11:57:26.980838+0000","last_clean":"2024-09-18T11:57:26.980838+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:26.980838+0000","last_undegraded":"2024-09-18T11:57:26.980838+0000","last_fullsized":"2024-09-18T11:57:26.980838+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8181,"log_dups_size":0,"ondisk_log_size":8181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33480704,"num_objects":8174,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5346,"num_write_kb":21384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8139","reported_seq":8220,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.986434+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:26.986434+0000","last_peered":"2024-09-18T11:57:26.986434+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:26.986434+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8139,"log_dups_size":0,"ondisk_log_size":8139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33318098,"num_objects":8136,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8137,"num_write_kb":32539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4609},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8127","reported_seq":8214,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.896646+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:26.896646+0000","last_peered":"2024-09-18T11:57:26.896646+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:26.896646+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264850,"num_objects":8123,"num_object_clones":0,"num_object_copies":16246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":32488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4564},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8245","reported_seq":8470,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:26.924111+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:26.924111+0000","last_peered":"2024-09-18T11:57:26.924111+0000","last_clean":"2024-09-18T11:57:26.924111+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:26.924111+0000","last_undegraded":"2024-09-18T11:57:26.924111+0000","last_fullsized":"2024-09-18T11:57:26.924111+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8245,"log_dups_size":0,"ondisk_log_size":8245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33763328,"num_objects":8243,"num_object_clones":0,"num_object_copies":16486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":18248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8135","reported_seq":15263,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:27.238394+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:27.238394+0000","last_peered":"2024-09-18T11:57:27.238394+0000","last_clean":"2024-09-18T11:57:27.238394+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:27.238394+0000","last_undegraded":"2024-09-18T11:57:27.238394+0000","last_fullsized":"2024-09-18T11:57:27.238394+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8135,"log_dups_size":0,"ondisk_log_size":8135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33293522,"num_objects":8130,"num_object_clones":0,"num_object_copies":16260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":32516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8137","reported_seq":8215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.933162+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:26.933162+0000","last_peered":"2024-09-18T11:57:26.933162+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:26.933162+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8137,"log_dups_size":0,"ondisk_log_size":8137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33297618,"num_objects":8131,"num_object_clones":0,"num_object_copies":16262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8135,"num_write_kb":32520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4599},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8187","reported_seq":8243,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:27.771175+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:27.771175+0000","last_peered":"2024-09-18T11:57:27.771175+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:27.771175+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8187,"log_dups_size":0,"ondisk_log_size":8187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33514706,"num_objects":8184,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8185,"num_write_kb":32731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8158","reported_seq":8367,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:27.741846+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:27.741846+0000","last_peered":"2024-09-18T11:57:27.741846+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:27.741846+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8158,"log_dups_size":0,"ondisk_log_size":8158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33390592,"num_objects":8152,"num_object_clones":0,"num_object_copies":16304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4638,"num_write_kb":18552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4559}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8137","reported_seq":8219,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.965167+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:26.965167+0000","last_peered":"2024-09-18T11:57:26.965167+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:26.965167+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8137,"log_dups_size":0,"ondisk_log_size":8137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33321577,"num_objects":8136,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8136,"num_write_kb":32541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4601},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8217","reported_seq":9534,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:27.017753+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:27.017753+0000","last_peered":"2024-09-18T11:57:27.017753+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:27.017753+0000","last_undegraded":"2024-09-18T11:57:27.017753+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33625915,"num_objects":8212,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2906,"num_objects_unfound":0,"num_objects_dirty":8212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":32840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":619,"num_bytes_recovered":2523136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5306},{"shards":"0,2","objects":2906}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8069","reported_seq":8147,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.998338+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:26.998338+0000","last_peered":"2024-09-18T11:57:26.998338+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:26.998338+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8069,"log_dups_size":0,"ondisk_log_size":8069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33035474,"num_objects":8067,"num_object_clones":0,"num_object_copies":16134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8067,"num_write_kb":32262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4474},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8210","reported_seq":8295,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:27.005100+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:27.005100+0000","last_peered":"2024-09-18T11:57:27.005100+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:27.005100+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8210,"log_dups_size":0,"ondisk_log_size":8210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33581476,"num_objects":8202,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":32799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4679},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":534741044,"num_objects":130569,"num_object_clones":0,"num_object_copies":261138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2906,"num_objects_unfound":0,"num_objects_dirty":130569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114624,"num_write_kb":458376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4119,"num_bytes_recovered":16781929,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1086533632,"data_stored":1086453615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130631,"ondisk_log_size":130631,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051627,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672888,"kb_used_data":375024,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93698952,"statfs":{"total":96636764160,"available":95947726848,"internally_reserved":0,"allocated":384024576,"data_stored":383801001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.189}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.26500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855390,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543504,"kb_used_data":237320,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93828336,"statfs":{"total":96636764160,"available":96080216064,"internally_reserved":0,"allocated":243015680,"data_stored":242718903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.313},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":3.6909999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.34999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753580,"kb_used_data":447460,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93618260,"statfs":{"total":96636764160,"available":95865098240,"internally_reserved":0,"allocated":458199040,"data_stored":457971843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.31900000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.38300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":458702848,"data_stored":458699369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243412992,"data_stored":243339933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":384417792,"data_stored":384414313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:31.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:31.935 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:31.954 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:31.961 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-18T11:57:31.087888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534741063,"num_objects":130571,"num_object_clones":0,"num_object_copies":261142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2906,"num_objects_unfound":0,"num_objects_dirty":130571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114626,"num_write_kb":458378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4120,"num_bytes_recovered":16781948,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130634,"ondisk_log_size":130634,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1969972,"kb_used_data":1059804,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281145548,"statfs":{"total":289910292480,"available":287893041152,"internally_reserved":0,"allocated":1085239296,"data_stored":1084491747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,1,2,5,9,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18321408,"num_objects":4473,"num_object_clones":0,"num_object_copies":8946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-200,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4473,"num_write_kb":17892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002818"},"pg_stats":[{"pgid":"2.f","version":"39'8165","reported_seq":8236,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.980275+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:26.980275+0000","last_peered":"2024-09-18T11:57:26.980275+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:26.980275+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8165,"log_dups_size":0,"ondisk_log_size":8165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33443840,"num_objects":8165,"num_object_clones":0,"num_object_copies":16330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8165,"num_write_kb":32660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4566},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8181","reported_seq":8339,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:27.707941+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:27.707941+0000","last_peered":"2024-09-18T11:57:27.707941+0000","last_clean":"2024-09-18T11:57:27.707941+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:27.707941+0000","last_undegraded":"2024-09-18T11:57:27.707941+0000","last_fullsized":"2024-09-18T11:57:27.707941+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8181,"log_dups_size":0,"ondisk_log_size":8181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33484800,"num_objects":8175,"num_object_clones":0,"num_object_copies":16350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5373,"num_write_kb":21492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8228","reported_seq":8413,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:27.723770+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:27.723770+0000","last_peered":"2024-09-18T11:57:27.723770+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:27.723770+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8228,"log_dups_size":0,"ondisk_log_size":8228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":8224,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5086,"num_write_kb":20344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4688}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8115","reported_seq":8144,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:27.745137+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:27.745137+0000","last_peered":"2024-09-18T11:57:27.745137+0000","last_clean":"2024-09-18T11:57:27.745137+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:27.745137+0000","last_undegraded":"2024-09-18T11:57:27.745137+0000","last_fullsized":"2024-09-18T11:57:27.745137+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8115,"log_dups_size":0,"ondisk_log_size":8115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33239040,"num_objects":8115,"num_object_clones":0,"num_object_copies":16230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8115,"num_write_kb":32460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8181","reported_seq":8372,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:26.980838+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:26.980838+0000","last_peered":"2024-09-18T11:57:26.980838+0000","last_clean":"2024-09-18T11:57:26.980838+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:26.980838+0000","last_undegraded":"2024-09-18T11:57:26.980838+0000","last_fullsized":"2024-09-18T11:57:26.980838+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8181,"log_dups_size":0,"ondisk_log_size":8181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33480704,"num_objects":8174,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5346,"num_write_kb":21384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8139","reported_seq":8220,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.986434+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:26.986434+0000","last_peered":"2024-09-18T11:57:26.986434+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:26.986434+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8139,"log_dups_size":0,"ondisk_log_size":8139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33318098,"num_objects":8136,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8137,"num_write_kb":32539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4609},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8127","reported_seq":8214,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.896646+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:26.896646+0000","last_peered":"2024-09-18T11:57:26.896646+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:26.896646+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33264850,"num_objects":8123,"num_object_clones":0,"num_object_copies":16246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":32488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4564},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8245","reported_seq":8470,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:26.924111+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:26.924111+0000","last_peered":"2024-09-18T11:57:26.924111+0000","last_clean":"2024-09-18T11:57:26.924111+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:26.924111+0000","last_undegraded":"2024-09-18T11:57:26.924111+0000","last_fullsized":"2024-09-18T11:57:26.924111+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8245,"log_dups_size":0,"ondisk_log_size":8245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33763328,"num_objects":8243,"num_object_clones":0,"num_object_copies":16486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":18248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8135","reported_seq":15263,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:27.238394+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:27.238394+0000","last_peered":"2024-09-18T11:57:27.238394+0000","last_clean":"2024-09-18T11:57:27.238394+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:27.238394+0000","last_undegraded":"2024-09-18T11:57:27.238394+0000","last_fullsized":"2024-09-18T11:57:27.238394+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8135,"log_dups_size":0,"ondisk_log_size":8135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33293522,"num_objects":8130,"num_object_clones":0,"num_object_copies":16260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":32516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8137","reported_seq":8215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.933162+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:26.933162+0000","last_peered":"2024-09-18T11:57:26.933162+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:26.933162+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8137,"log_dups_size":0,"ondisk_log_size":8137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33297618,"num_objects":8131,"num_object_clones":0,"num_object_copies":16262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8135,"num_write_kb":32520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4599},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8187","reported_seq":8243,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:27.771175+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:27.771175+0000","last_peered":"2024-09-18T11:57:27.771175+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:27.771175+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8187,"log_dups_size":0,"ondisk_log_size":8187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33514706,"num_objects":8184,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8185,"num_write_kb":32731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8158","reported_seq":8367,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:27.741846+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:27.741846+0000","last_peered":"2024-09-18T11:57:27.741846+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:27.741846+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8158,"log_dups_size":0,"ondisk_log_size":8158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33390592,"num_objects":8152,"num_object_clones":0,"num_object_copies":16304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4638,"num_write_kb":18552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4559}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8137","reported_seq":8219,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.965167+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:26.965167+0000","last_peered":"2024-09-18T11:57:26.965167+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:26.965167+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8137,"log_dups_size":0,"ondisk_log_size":8137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33321577,"num_objects":8136,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8136,"num_write_kb":32541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4601},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8217","reported_seq":9534,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:27.017753+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:27.017753+0000","last_peered":"2024-09-18T11:57:27.017753+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:27.017753+0000","last_undegraded":"2024-09-18T11:57:27.017753+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33625915,"num_objects":8212,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2906,"num_objects_unfound":0,"num_objects_dirty":8212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":32840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":619,"num_bytes_recovered":2523136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5306},{"shards":"0,2","objects":2906}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8069","reported_seq":8147,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:26.998338+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:26.998338+0000","last_peered":"2024-09-18T11:57:26.998338+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:26.998338+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8069,"log_dups_size":0,"ondisk_log_size":8069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33035474,"num_objects":8067,"num_object_clones":0,"num_object_copies":16134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8067,"num_write_kb":32262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4474},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8210","reported_seq":8295,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:27.005100+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:27.005100+0000","last_peered":"2024-09-18T11:57:27.005100+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:27.005100+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8210,"log_dups_size":0,"ondisk_log_size":8210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33581476,"num_objects":8202,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":32799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4679},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":534741044,"num_objects":130569,"num_object_clones":0,"num_object_copies":261138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2906,"num_objects_unfound":0,"num_objects_dirty":130569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114624,"num_write_kb":458376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4119,"num_bytes_recovered":16781929,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1086533632,"data_stored":1086453615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130631,"ondisk_log_size":130631,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051627,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":672888,"kb_used_data":375024,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93698952,"statfs":{"total":96636764160,"available":95947726848,"internally_reserved":0,"allocated":384024576,"data_stored":383801001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.189}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.26500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855390,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543504,"kb_used_data":237320,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93828336,"statfs":{"total":96636764160,"available":96080216064,"internally_reserved":0,"allocated":243015680,"data_stored":242718903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.313},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":3.6909999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.34999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753580,"kb_used_data":447460,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93618260,"statfs":{"total":96636764160,"available":95865098240,"internally_reserved":0,"allocated":458199040,"data_stored":457971843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.31900000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.38300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":458702848,"data_stored":458699369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":243412992,"data_stored":243339933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":384417792,"data_stored":384414313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:31.962 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:57:31.962 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'8165", 'reported_seq': 8236, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:26.980275+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:57:26.980275+0000', 'last_peered': '2024-09-18T11:57:26.980275+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:57:26.980275+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8165, 'log_dups_size': 0, 'ondisk_log_size': 8165, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33443840, 'num_objects': 8165, 'num_object_clones': 0, 'num_object_copies': 16330, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8165, 'num_write_kb': 32660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4566}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:31.962 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:57:31.962 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'8228", 'reported_seq': 8413, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:27.723770+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:57:27.723770+0000', 'last_peered': '2024-09-18T11:57:27.723770+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:57:27.723770+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 8228, 'log_dups_size': 0, 'ondisk_log_size': 8228, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33685504, 'num_objects': 8224, 'num_object_clones': 0, 'num_object_copies': 16448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8224, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5086, 'num_write_kb': 20344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 4688}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:31.963 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:57:31.963 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'8139", 'reported_seq': 8220, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:26.986434+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:57:26.986434+0000', 'last_peered': '2024-09-18T11:57:26.986434+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:57:26.986434+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8139, 'log_dups_size': 0, 'ondisk_log_size': 8139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33318098, 'num_objects': 8136, 'num_object_clones': 0, 'num_object_copies': 16272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8137, 'num_write_kb': 32539, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4609}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:31.963 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:57:31.963 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'8127", 'reported_seq': 8214, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:26.896646+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:57:26.896646+0000', 'last_peered': '2024-09-18T11:57:26.896646+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:57:26.896646+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8127, 'log_dups_size': 0, 'ondisk_log_size': 8127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33264850, 'num_objects': 8123, 'num_object_clones': 0, 'num_object_copies': 16246, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8125, 'num_write_kb': 32488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4564}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:31.963 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:57:31.963 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'8137", 'reported_seq': 8215, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:26.933162+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:57:26.933162+0000', 'last_peered': '2024-09-18T11:57:26.933162+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:57:26.933162+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8137, 'log_dups_size': 0, 'ondisk_log_size': 8137, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33297618, 'num_objects': 8131, 'num_object_clones': 0, 'num_object_copies': 16262, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8135, 'num_write_kb': 32520, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4599}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:31.963 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:57:31.963 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'8187", 'reported_seq': 8243, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:27.771175+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:57:27.771175+0000', 'last_peered': '2024-09-18T11:57:27.771175+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:57:27.771175+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8187, 'log_dups_size': 0, 'ondisk_log_size': 8187, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33514706, 'num_objects': 8184, 'num_object_clones': 0, 'num_object_copies': 16368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8185, 'num_write_kb': 32731, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 4634}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:31.963 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:57:31.963 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'8158", 'reported_seq': 8367, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:27.741846+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:57:27.741846+0000', 'last_peered': '2024-09-18T11:57:27.741846+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:57:27.741846+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 8158, 'log_dups_size': 0, 'ondisk_log_size': 8158, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33390592, 'num_objects': 8152, 'num_object_clones': 0, 'num_object_copies': 16304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4638, 'num_write_kb': 18552, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 4559}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:31.963 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:57:31.964 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'8137", 'reported_seq': 8219, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:26.965167+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:57:26.965167+0000', 'last_peered': '2024-09-18T11:57:26.965167+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:57:26.965167+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8137, 'log_dups_size': 0, 'ondisk_log_size': 8137, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33321577, 'num_objects': 8136, 'num_object_clones': 0, 'num_object_copies': 16272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8136, 'num_write_kb': 32541, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4601}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:31.964 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:57:31.964 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'8217", 'reported_seq': 9534, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:57:27.017753+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:57:27.017753+0000', 'last_peered': '2024-09-18T11:57:27.017753+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:57:27.017753+0000', 'last_undegraded': '2024-09-18T11:57:27.017753+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8217, 'log_dups_size': 0, 'ondisk_log_size': 8217, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33625915, 'num_objects': 8212, 'num_object_clones': 0, 'num_object_copies': 16424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2906, 'num_objects_unfound': 0, 'num_objects_dirty': 8212, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8214, 'num_write_kb': 32840, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 619, 'num_bytes_recovered': 2523136, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5306}, {'shards': '0,2', 'objects': 2906}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:31.964 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:57:31.964 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'8069", 'reported_seq': 8147, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:26.998338+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:57:26.998338+0000', 'last_peered': '2024-09-18T11:57:26.998338+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:57:26.998338+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8069, 'log_dups_size': 0, 'ondisk_log_size': 8069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33035474, 'num_objects': 8067, 'num_object_clones': 0, 'num_object_copies': 16134, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8067, 'num_write_kb': 32262, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4474}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:31.964 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:57:31.964 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'8210", 'reported_seq': 8295, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:27.005100+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:57:27.005100+0000', 'last_peered': '2024-09-18T11:57:27.005100+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:57:27.005100+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8210, 'log_dups_size': 0, 'ondisk_log_size': 8210, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33581476, 'num_objects': 8202, 'num_object_clones': 0, 'num_object_copies': 16404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8207, 'num_write_kb': 32799, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4679}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:31.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:32.148 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 186 20 132743 132723 2.7865 1.62891 0.0275993 0.0280269 2024-09-18T11:57:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:32.812 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/261142 objects degraded (13.619%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":252,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":130571,"data_bytes":534741063,"bytes_used":2017251328,"bytes_avail":287893041152,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":261142,"degraded_ratio":0.1361864426250852,"misplaced_objects":2906,"misplaced_total":261142,"misplaced_ratio":0.011128045278047959,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":68250,"recovering_keys_per_sec":0,"num_objects_recovered":200,"num_bytes_recovered":819200,"num_keys_recovered":0,"write_bytes_sec":1526425,"read_op_per_sec":0,"write_op_per_sec":372},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:32.845 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:32.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:33.148 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 187 20 133169 133149 2.7805 1.66406 0.0514049 0.0280894 2024-09-18T11:57:33.705 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:33.742 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:33.749 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-18T11:57:33.088332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":543969351,"num_objects":132824,"num_object_clones":0,"num_object_copies":265648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2802,"num_objects_unfound":0,"num_objects_dirty":132824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116879,"num_write_kb":467390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4224,"num_bytes_recovered":17207932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132887,"ondisk_log_size":132887,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2000716,"kb_used_data":1078068,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281114804,"statfs":{"total":289910292480,"available":287861559296,"internally_reserved":0,"allocated":1103941632,"data_stored":1103194083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,4,3,13,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27549696,"num_objects":6726,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-304,"num_objects_unfound":0,"num_objects_dirty":6726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6726,"num_write_kb":26904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":304,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002790"},"pg_stats":[{"pgid":"2.f","version":"39'8308","reported_seq":8379,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:31.946730+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:31.946730+0000","last_peered":"2024-09-18T11:57:31.946730+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:31.946730+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34029568,"num_objects":8308,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":33232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4709},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8313","reported_seq":8470,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:32.800582+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:32.800582+0000","last_peered":"2024-09-18T11:57:32.800582+0000","last_clean":"2024-09-18T11:57:32.800582+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:32.800582+0000","last_undegraded":"2024-09-18T11:57:32.800582+0000","last_fullsized":"2024-09-18T11:57:32.800582+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8313,"log_dups_size":0,"ondisk_log_size":8313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34025472,"num_objects":8307,"num_object_clones":0,"num_object_copies":16614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5505,"num_write_kb":22020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8391","reported_seq":8574,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.756275+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:32.756275+0000","last_peered":"2024-09-18T11:57:32.756275+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:32.756275+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34353152,"num_objects":8387,"num_object_clones":0,"num_object_copies":16774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5249,"num_write_kb":20996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8242","reported_seq":8271,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:32.709598+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:32.709598+0000","last_peered":"2024-09-18T11:57:32.709598+0000","last_clean":"2024-09-18T11:57:32.709598+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:32.709598+0000","last_undegraded":"2024-09-18T11:57:32.709598+0000","last_fullsized":"2024-09-18T11:57:32.709598+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8242,"log_dups_size":0,"ondisk_log_size":8242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33759232,"num_objects":8242,"num_object_clones":0,"num_object_copies":16484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8242,"num_write_kb":32968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8330","reported_seq":8521,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:31.927229+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:31.927229+0000","last_peered":"2024-09-18T11:57:31.927229+0000","last_clean":"2024-09-18T11:57:31.927229+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:31.927229+0000","last_undegraded":"2024-09-18T11:57:31.927229+0000","last_fullsized":"2024-09-18T11:57:31.927229+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8330,"log_dups_size":0,"ondisk_log_size":8330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34091008,"num_objects":8323,"num_object_clones":0,"num_object_copies":16646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":21980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8274","reported_seq":8355,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:31.955192+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:31.955192+0000","last_peered":"2024-09-18T11:57:31.955192+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:31.955192+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8274,"log_dups_size":0,"ondisk_log_size":8274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33871058,"num_objects":8271,"num_object_clones":0,"num_object_copies":16542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8272,"num_write_kb":33079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4744},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8244","reported_seq":8331,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.040614+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:32.040614+0000","last_peered":"2024-09-18T11:57:32.040614+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:32.040614+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33744082,"num_objects":8240,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8242,"num_write_kb":32956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4681},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8392","reported_seq":8617,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:32.018800+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:32.018800+0000","last_peered":"2024-09-18T11:57:32.018800+0000","last_clean":"2024-09-18T11:57:32.018800+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:32.018800+0000","last_undegraded":"2024-09-18T11:57:32.018800+0000","last_fullsized":"2024-09-18T11:57:32.018800+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8392,"log_dups_size":0,"ondisk_log_size":8392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34365440,"num_objects":8390,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":18836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8278","reported_seq":15406,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:32.227368+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:32.227368+0000","last_peered":"2024-09-18T11:57:32.227368+0000","last_clean":"2024-09-18T11:57:32.227368+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:32.227368+0000","last_undegraded":"2024-09-18T11:57:32.227368+0000","last_fullsized":"2024-09-18T11:57:32.227368+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33879250,"num_objects":8273,"num_object_clones":0,"num_object_copies":16546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8276,"num_write_kb":33088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8281","reported_seq":8355,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:31.912142+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:31.912142+0000","last_peered":"2024-09-18T11:57:31.912142+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:31.912142+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33887442,"num_objects":8275,"num_object_clones":0,"num_object_copies":16550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8279,"num_write_kb":33096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4743},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8330","reported_seq":8386,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.734359+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:32.734359+0000","last_peered":"2024-09-18T11:57:32.734359+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:32.734359+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8330,"log_dups_size":0,"ondisk_log_size":8330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34100434,"num_objects":8327,"num_object_clones":0,"num_object_copies":16654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":33303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4777}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8308","reported_seq":8517,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.668367+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:32.668367+0000","last_peered":"2024-09-18T11:57:32.668367+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:32.668367+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34004992,"num_objects":8302,"num_object_clones":0,"num_object_copies":16604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4788,"num_write_kb":19152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4709}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8273","reported_seq":8355,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:31.988799+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:31.988799+0000","last_peered":"2024-09-18T11:57:31.988799+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:31.988799+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33878633,"num_objects":8272,"num_object_clones":0,"num_object_copies":16544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8272,"num_write_kb":33085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4737},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8356","reported_seq":9880,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:31.990578+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:31.990578+0000","last_peered":"2024-09-18T11:57:31.990578+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:31.990578+0000","last_undegraded":"2024-09-18T11:57:31.990578+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8356,"log_dups_size":0,"ondisk_log_size":8356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34195259,"num_objects":8351,"num_object_clones":0,"num_object_copies":16702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2802,"num_objects_unfound":0,"num_objects_dirty":8351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":33396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":723,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5549},{"shards":"0,2","objects":2802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8201","reported_seq":8282,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.053191+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:32.053191+0000","last_peered":"2024-09-18T11:57:32.053191+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:32.053191+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33576146,"num_objects":8199,"num_object_clones":0,"num_object_copies":16398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8199,"num_write_kb":32790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4606},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8363","reported_seq":8448,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.019467+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:32.019467+0000","last_peered":"2024-09-18T11:57:32.019467+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:32.019467+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8363,"log_dups_size":0,"ondisk_log_size":8363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34208164,"num_objects":8355,"num_object_clones":0,"num_object_copies":16710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8360,"num_write_kb":33411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4832},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":543969332,"num_objects":132822,"num_object_clones":0,"num_object_copies":265644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2802,"num_objects_unfound":0,"num_objects_dirty":132822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116877,"num_write_kb":467388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4223,"num_bytes_recovered":17207913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1105317888,"data_stored":1105237871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132884,"ondisk_log_size":132884,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051628,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681532,"kb_used_data":379508,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93690308,"statfs":{"total":96636764160,"available":95938875392,"internally_reserved":0,"allocated":388616192,"data_stored":388392617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":1.4910000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.56799999999999995}]}]},{"osd":1,"up_from":35,"seq":150323855391,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554952,"kb_used_data":244608,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93816888,"statfs":{"total":96636764160,"available":96068493312,"internally_reserved":0,"allocated":250478592,"data_stored":250181815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.23699999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.437},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.253}]}]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764232,"kb_used_data":453952,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93607608,"statfs":{"total":96636764160,"available":95854190592,"internally_reserved":0,"allocated":464846848,"data_stored":464619651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.17599999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.23300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":465444864,"data_stored":465441385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":250773504,"data_stored":250700445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":389099520,"data_stored":389096041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:34.149 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 188 20 133611 133591 2.77489 1.72656 0.0258399 0.0281447 2024-09-18T11:57:35.149 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 189 20 134069 134049 2.76968 1.78906 0.0550002 0.0282 2024-09-18T11:57:36.149 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 190 20 134524 134504 2.76445 1.77734 0.0156033 0.028253 2024-09-18T11:57:36.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:37.149 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 191 20 134979 134959 2.75929 1.77734 0.0301237 0.028305 2024-09-18T11:57:37.326 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:37.358 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:37.367 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-18T11:57:37.089318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":543969351,"num_objects":132824,"num_object_clones":0,"num_object_copies":265648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2802,"num_objects_unfound":0,"num_objects_dirty":132824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116879,"num_write_kb":467390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4224,"num_bytes_recovered":17207932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132887,"ondisk_log_size":132887,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2000716,"kb_used_data":1078068,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281114804,"statfs":{"total":289910292480,"available":287861559296,"internally_reserved":0,"allocated":1103941632,"data_stored":1103194083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,4,3,13,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18329600,"num_objects":4475,"num_object_clones":0,"num_object_copies":8950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-196,"num_objects_unfound":0,"num_objects_dirty":4475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4475,"num_write_kb":17900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002830"},"pg_stats":[{"pgid":"2.f","version":"39'8308","reported_seq":8379,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:31.946730+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:31.946730+0000","last_peered":"2024-09-18T11:57:31.946730+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:31.946730+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34029568,"num_objects":8308,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":33232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4709},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8313","reported_seq":8470,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:32.800582+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:32.800582+0000","last_peered":"2024-09-18T11:57:32.800582+0000","last_clean":"2024-09-18T11:57:32.800582+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:32.800582+0000","last_undegraded":"2024-09-18T11:57:32.800582+0000","last_fullsized":"2024-09-18T11:57:32.800582+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8313,"log_dups_size":0,"ondisk_log_size":8313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34025472,"num_objects":8307,"num_object_clones":0,"num_object_copies":16614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5505,"num_write_kb":22020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8391","reported_seq":8574,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.756275+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:32.756275+0000","last_peered":"2024-09-18T11:57:32.756275+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:32.756275+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34353152,"num_objects":8387,"num_object_clones":0,"num_object_copies":16774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5249,"num_write_kb":20996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8242","reported_seq":8271,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:32.709598+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:32.709598+0000","last_peered":"2024-09-18T11:57:32.709598+0000","last_clean":"2024-09-18T11:57:32.709598+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:32.709598+0000","last_undegraded":"2024-09-18T11:57:32.709598+0000","last_fullsized":"2024-09-18T11:57:32.709598+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8242,"log_dups_size":0,"ondisk_log_size":8242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33759232,"num_objects":8242,"num_object_clones":0,"num_object_copies":16484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8242,"num_write_kb":32968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8330","reported_seq":8521,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:31.927229+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:31.927229+0000","last_peered":"2024-09-18T11:57:31.927229+0000","last_clean":"2024-09-18T11:57:31.927229+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:31.927229+0000","last_undegraded":"2024-09-18T11:57:31.927229+0000","last_fullsized":"2024-09-18T11:57:31.927229+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8330,"log_dups_size":0,"ondisk_log_size":8330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34091008,"num_objects":8323,"num_object_clones":0,"num_object_copies":16646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":21980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8274","reported_seq":8355,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:31.955192+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:31.955192+0000","last_peered":"2024-09-18T11:57:31.955192+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:31.955192+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8274,"log_dups_size":0,"ondisk_log_size":8274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33871058,"num_objects":8271,"num_object_clones":0,"num_object_copies":16542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8272,"num_write_kb":33079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4744},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8244","reported_seq":8331,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.040614+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:32.040614+0000","last_peered":"2024-09-18T11:57:32.040614+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:32.040614+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33744082,"num_objects":8240,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8242,"num_write_kb":32956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4681},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8392","reported_seq":8617,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:32.018800+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:32.018800+0000","last_peered":"2024-09-18T11:57:32.018800+0000","last_clean":"2024-09-18T11:57:32.018800+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:32.018800+0000","last_undegraded":"2024-09-18T11:57:32.018800+0000","last_fullsized":"2024-09-18T11:57:32.018800+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8392,"log_dups_size":0,"ondisk_log_size":8392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34365440,"num_objects":8390,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":18836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8278","reported_seq":15406,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:32.227368+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:32.227368+0000","last_peered":"2024-09-18T11:57:32.227368+0000","last_clean":"2024-09-18T11:57:32.227368+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:32.227368+0000","last_undegraded":"2024-09-18T11:57:32.227368+0000","last_fullsized":"2024-09-18T11:57:32.227368+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33879250,"num_objects":8273,"num_object_clones":0,"num_object_copies":16546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8276,"num_write_kb":33088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8281","reported_seq":8355,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:31.912142+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:31.912142+0000","last_peered":"2024-09-18T11:57:31.912142+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:31.912142+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33887442,"num_objects":8275,"num_object_clones":0,"num_object_copies":16550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8279,"num_write_kb":33096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4743},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8330","reported_seq":8386,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.734359+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:32.734359+0000","last_peered":"2024-09-18T11:57:32.734359+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:32.734359+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8330,"log_dups_size":0,"ondisk_log_size":8330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34100434,"num_objects":8327,"num_object_clones":0,"num_object_copies":16654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":33303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4777}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8308","reported_seq":8517,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.668367+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:32.668367+0000","last_peered":"2024-09-18T11:57:32.668367+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:32.668367+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34004992,"num_objects":8302,"num_object_clones":0,"num_object_copies":16604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4788,"num_write_kb":19152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4709}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8273","reported_seq":8355,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:31.988799+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:31.988799+0000","last_peered":"2024-09-18T11:57:31.988799+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:31.988799+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33878633,"num_objects":8272,"num_object_clones":0,"num_object_copies":16544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8272,"num_write_kb":33085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4737},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8356","reported_seq":9880,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:31.990578+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:31.990578+0000","last_peered":"2024-09-18T11:57:31.990578+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:31.990578+0000","last_undegraded":"2024-09-18T11:57:31.990578+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8356,"log_dups_size":0,"ondisk_log_size":8356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34195259,"num_objects":8351,"num_object_clones":0,"num_object_copies":16702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2802,"num_objects_unfound":0,"num_objects_dirty":8351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":33396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":723,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5549},{"shards":"0,2","objects":2802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8201","reported_seq":8282,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.053191+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:32.053191+0000","last_peered":"2024-09-18T11:57:32.053191+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:32.053191+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33576146,"num_objects":8199,"num_object_clones":0,"num_object_copies":16398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8199,"num_write_kb":32790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4606},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8363","reported_seq":8448,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.019467+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:32.019467+0000","last_peered":"2024-09-18T11:57:32.019467+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:32.019467+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8363,"log_dups_size":0,"ondisk_log_size":8363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34208164,"num_objects":8355,"num_object_clones":0,"num_object_copies":16710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8360,"num_write_kb":33411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4832},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":543969332,"num_objects":132822,"num_object_clones":0,"num_object_copies":265644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2802,"num_objects_unfound":0,"num_objects_dirty":132822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116877,"num_write_kb":467388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4223,"num_bytes_recovered":17207913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1105317888,"data_stored":1105237871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132884,"ondisk_log_size":132884,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051628,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681532,"kb_used_data":379508,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93690308,"statfs":{"total":96636764160,"available":95938875392,"internally_reserved":0,"allocated":388616192,"data_stored":388392617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":1.4910000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.56799999999999995}]}]},{"osd":1,"up_from":35,"seq":150323855391,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554952,"kb_used_data":244608,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93816888,"statfs":{"total":96636764160,"available":96068493312,"internally_reserved":0,"allocated":250478592,"data_stored":250181815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.23699999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.437},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.253}]}]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764232,"kb_used_data":453952,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93607608,"statfs":{"total":96636764160,"available":95854190592,"internally_reserved":0,"allocated":464846848,"data_stored":464619651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.17599999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.23300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":465444864,"data_stored":465441385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":250773504,"data_stored":250700445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":389099520,"data_stored":389096041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:37.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:38.083 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:38.087 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:38.116 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-18T11:57:37.089318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":543969351,"num_objects":132824,"num_object_clones":0,"num_object_copies":265648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2802,"num_objects_unfound":0,"num_objects_dirty":132824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116879,"num_write_kb":467390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4224,"num_bytes_recovered":17207932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132887,"ondisk_log_size":132887,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2000716,"kb_used_data":1078068,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281114804,"statfs":{"total":289910292480,"available":287861559296,"internally_reserved":0,"allocated":1103941632,"data_stored":1103194083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,4,3,13,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18329600,"num_objects":4475,"num_object_clones":0,"num_object_copies":8950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-196,"num_objects_unfound":0,"num_objects_dirty":4475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4475,"num_write_kb":17900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002830"},"pg_stats":[{"pgid":"2.f","version":"39'8308","reported_seq":8379,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:31.946730+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:31.946730+0000","last_peered":"2024-09-18T11:57:31.946730+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:31.946730+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34029568,"num_objects":8308,"num_object_clones":0,"num_object_copies":16616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":33232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4709},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8313","reported_seq":8470,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:32.800582+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:32.800582+0000","last_peered":"2024-09-18T11:57:32.800582+0000","last_clean":"2024-09-18T11:57:32.800582+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:32.800582+0000","last_undegraded":"2024-09-18T11:57:32.800582+0000","last_fullsized":"2024-09-18T11:57:32.800582+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8313,"log_dups_size":0,"ondisk_log_size":8313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34025472,"num_objects":8307,"num_object_clones":0,"num_object_copies":16614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5505,"num_write_kb":22020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8391","reported_seq":8574,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.756275+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:32.756275+0000","last_peered":"2024-09-18T11:57:32.756275+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:32.756275+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34353152,"num_objects":8387,"num_object_clones":0,"num_object_copies":16774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5249,"num_write_kb":20996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8242","reported_seq":8271,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:32.709598+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:32.709598+0000","last_peered":"2024-09-18T11:57:32.709598+0000","last_clean":"2024-09-18T11:57:32.709598+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:32.709598+0000","last_undegraded":"2024-09-18T11:57:32.709598+0000","last_fullsized":"2024-09-18T11:57:32.709598+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8242,"log_dups_size":0,"ondisk_log_size":8242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33759232,"num_objects":8242,"num_object_clones":0,"num_object_copies":16484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8242,"num_write_kb":32968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8330","reported_seq":8521,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:31.927229+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:31.927229+0000","last_peered":"2024-09-18T11:57:31.927229+0000","last_clean":"2024-09-18T11:57:31.927229+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:31.927229+0000","last_undegraded":"2024-09-18T11:57:31.927229+0000","last_fullsized":"2024-09-18T11:57:31.927229+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8330,"log_dups_size":0,"ondisk_log_size":8330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34091008,"num_objects":8323,"num_object_clones":0,"num_object_copies":16646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":21980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8274","reported_seq":8355,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:31.955192+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:31.955192+0000","last_peered":"2024-09-18T11:57:31.955192+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:31.955192+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8274,"log_dups_size":0,"ondisk_log_size":8274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33871058,"num_objects":8271,"num_object_clones":0,"num_object_copies":16542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8272,"num_write_kb":33079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4744},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8244","reported_seq":8331,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.040614+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:32.040614+0000","last_peered":"2024-09-18T11:57:32.040614+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:32.040614+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33744082,"num_objects":8240,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8242,"num_write_kb":32956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4681},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8392","reported_seq":8617,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:32.018800+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:32.018800+0000","last_peered":"2024-09-18T11:57:32.018800+0000","last_clean":"2024-09-18T11:57:32.018800+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:32.018800+0000","last_undegraded":"2024-09-18T11:57:32.018800+0000","last_fullsized":"2024-09-18T11:57:32.018800+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8392,"log_dups_size":0,"ondisk_log_size":8392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34365440,"num_objects":8390,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":18836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8278","reported_seq":15406,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:32.227368+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:32.227368+0000","last_peered":"2024-09-18T11:57:32.227368+0000","last_clean":"2024-09-18T11:57:32.227368+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:32.227368+0000","last_undegraded":"2024-09-18T11:57:32.227368+0000","last_fullsized":"2024-09-18T11:57:32.227368+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33879250,"num_objects":8273,"num_object_clones":0,"num_object_copies":16546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8276,"num_write_kb":33088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8281","reported_seq":8355,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:31.912142+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:31.912142+0000","last_peered":"2024-09-18T11:57:31.912142+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:31.912142+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33887442,"num_objects":8275,"num_object_clones":0,"num_object_copies":16550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8279,"num_write_kb":33096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4743},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8330","reported_seq":8386,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.734359+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:32.734359+0000","last_peered":"2024-09-18T11:57:32.734359+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:32.734359+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8330,"log_dups_size":0,"ondisk_log_size":8330,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34100434,"num_objects":8327,"num_object_clones":0,"num_object_copies":16654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":33303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4777}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8308","reported_seq":8517,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.668367+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:32.668367+0000","last_peered":"2024-09-18T11:57:32.668367+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:32.668367+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34004992,"num_objects":8302,"num_object_clones":0,"num_object_copies":16604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4788,"num_write_kb":19152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4709}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8273","reported_seq":8355,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:31.988799+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:31.988799+0000","last_peered":"2024-09-18T11:57:31.988799+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:31.988799+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8273,"log_dups_size":0,"ondisk_log_size":8273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33878633,"num_objects":8272,"num_object_clones":0,"num_object_copies":16544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8272,"num_write_kb":33085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4737},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8356","reported_seq":9880,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:31.990578+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:31.990578+0000","last_peered":"2024-09-18T11:57:31.990578+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:31.990578+0000","last_undegraded":"2024-09-18T11:57:31.990578+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8356,"log_dups_size":0,"ondisk_log_size":8356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34195259,"num_objects":8351,"num_object_clones":0,"num_object_copies":16702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2802,"num_objects_unfound":0,"num_objects_dirty":8351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":33396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":723,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5549},{"shards":"0,2","objects":2802}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8201","reported_seq":8282,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.053191+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:32.053191+0000","last_peered":"2024-09-18T11:57:32.053191+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:32.053191+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33576146,"num_objects":8199,"num_object_clones":0,"num_object_copies":16398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8199,"num_write_kb":32790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4606},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8363","reported_seq":8448,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:32.019467+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:32.019467+0000","last_peered":"2024-09-18T11:57:32.019467+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:32.019467+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8363,"log_dups_size":0,"ondisk_log_size":8363,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34208164,"num_objects":8355,"num_object_clones":0,"num_object_copies":16710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8360,"num_write_kb":33411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4832},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":543969332,"num_objects":132822,"num_object_clones":0,"num_object_copies":265644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2802,"num_objects_unfound":0,"num_objects_dirty":132822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116877,"num_write_kb":467388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4223,"num_bytes_recovered":17207913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1105317888,"data_stored":1105237871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132884,"ondisk_log_size":132884,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051628,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681532,"kb_used_data":379508,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93690308,"statfs":{"total":96636764160,"available":95938875392,"internally_reserved":0,"allocated":388616192,"data_stored":388392617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":1.4910000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.56799999999999995}]}]},{"osd":1,"up_from":35,"seq":150323855391,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554952,"kb_used_data":244608,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93816888,"statfs":{"total":96636764160,"available":96068493312,"internally_reserved":0,"allocated":250478592,"data_stored":250181815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.23699999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.437},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.253}]}]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":764232,"kb_used_data":453952,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93607608,"statfs":{"total":96636764160,"available":95854190592,"internally_reserved":0,"allocated":464846848,"data_stored":464619651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.17599999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.23300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":465444864,"data_stored":465441385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":250773504,"data_stored":250700445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":389099520,"data_stored":389096041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:38.117 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:57:38.117 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'8308", 'reported_seq': 8379, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:31.946730+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:57:31.946730+0000', 'last_peered': '2024-09-18T11:57:31.946730+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:57:31.946730+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8308, 'log_dups_size': 0, 'ondisk_log_size': 8308, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34029568, 'num_objects': 8308, 'num_object_clones': 0, 'num_object_copies': 16616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8308, 'num_write_kb': 33232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4709}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:38.118 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:57:38.118 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'8391", 'reported_seq': 8574, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:32.756275+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:57:32.756275+0000', 'last_peered': '2024-09-18T11:57:32.756275+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:57:32.756275+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 8391, 'log_dups_size': 0, 'ondisk_log_size': 8391, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34353152, 'num_objects': 8387, 'num_object_clones': 0, 'num_object_copies': 16774, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5249, 'num_write_kb': 20996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 4851}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:38.118 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:57:38.118 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'8274", 'reported_seq': 8355, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:31.955192+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:57:31.955192+0000', 'last_peered': '2024-09-18T11:57:31.955192+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:57:31.955192+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8274, 'log_dups_size': 0, 'ondisk_log_size': 8274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33871058, 'num_objects': 8271, 'num_object_clones': 0, 'num_object_copies': 16542, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8271, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8272, 'num_write_kb': 33079, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4744}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:38.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:57:38.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'8244", 'reported_seq': 8331, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:32.040614+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:57:32.040614+0000', 'last_peered': '2024-09-18T11:57:32.040614+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:57:32.040614+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8244, 'log_dups_size': 0, 'ondisk_log_size': 8244, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33744082, 'num_objects': 8240, 'num_object_clones': 0, 'num_object_copies': 16480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8242, 'num_write_kb': 32956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4681}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:38.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:57:38.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'8281", 'reported_seq': 8355, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:31.912142+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:57:31.912142+0000', 'last_peered': '2024-09-18T11:57:31.912142+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:57:31.912142+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8281, 'log_dups_size': 0, 'ondisk_log_size': 8281, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33887442, 'num_objects': 8275, 'num_object_clones': 0, 'num_object_copies': 16550, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8275, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8279, 'num_write_kb': 33096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4743}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:38.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:57:38.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'8330", 'reported_seq': 8386, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:32.734359+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:57:32.734359+0000', 'last_peered': '2024-09-18T11:57:32.734359+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:57:32.734359+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8330, 'log_dups_size': 0, 'ondisk_log_size': 8330, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34100434, 'num_objects': 8327, 'num_object_clones': 0, 'num_object_copies': 16654, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8327, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8328, 'num_write_kb': 33303, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 4777}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:38.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:57:38.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'8308", 'reported_seq': 8517, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:32.668367+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:57:32.668367+0000', 'last_peered': '2024-09-18T11:57:32.668367+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:57:32.668367+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 8308, 'log_dups_size': 0, 'ondisk_log_size': 8308, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34004992, 'num_objects': 8302, 'num_object_clones': 0, 'num_object_copies': 16604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8302, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4788, 'num_write_kb': 19152, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 4709}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:38.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:57:38.119 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'8273", 'reported_seq': 8355, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:31.988799+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:57:31.988799+0000', 'last_peered': '2024-09-18T11:57:31.988799+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:57:31.988799+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8273, 'log_dups_size': 0, 'ondisk_log_size': 8273, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33878633, 'num_objects': 8272, 'num_object_clones': 0, 'num_object_copies': 16544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8272, 'num_write_kb': 33085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4737}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:38.120 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:57:38.120 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'8356", 'reported_seq': 9880, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:57:31.990578+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:57:31.990578+0000', 'last_peered': '2024-09-18T11:57:31.990578+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:57:31.990578+0000', 'last_undegraded': '2024-09-18T11:57:31.990578+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8356, 'log_dups_size': 0, 'ondisk_log_size': 8356, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34195259, 'num_objects': 8351, 'num_object_clones': 0, 'num_object_copies': 16702, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2802, 'num_objects_unfound': 0, 'num_objects_dirty': 8351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8353, 'num_write_kb': 33396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 723, 'num_bytes_recovered': 2949120, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5549}, {'shards': '0,2', 'objects': 2802}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:38.120 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:57:38.120 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'8201", 'reported_seq': 8282, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:32.053191+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:57:32.053191+0000', 'last_peered': '2024-09-18T11:57:32.053191+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:57:32.053191+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8201, 'log_dups_size': 0, 'ondisk_log_size': 8201, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33576146, 'num_objects': 8199, 'num_object_clones': 0, 'num_object_copies': 16398, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8199, 'num_write_kb': 32790, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4606}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:38.120 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:57:38.120 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'8363", 'reported_seq': 8448, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:32.019467+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:57:32.019467+0000', 'last_peered': '2024-09-18T11:57:32.019467+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:57:32.019467+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8363, 'log_dups_size': 0, 'ondisk_log_size': 8363, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34208164, 'num_objects': 8355, 'num_object_clones': 0, 'num_object_copies': 16710, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8360, 'num_write_kb': 33411, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4832}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:38.120 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:38.149 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 192 20 135396 135376 2.7534 1.62891 0.0390993 0.028366 2024-09-18T11:57:38.794 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:38.794 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/265648 objects degraded (13.388%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":258,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":132824,"data_bytes":543969351,"bytes_used":2048733184,"bytes_avail":287861559296,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":265648,"degraded_ratio":0.13387640787809432,"misplaced_objects":2802,"misplaced_total":265648,"misplaced_ratio":0.010547792567608263,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":66885,"recovering_keys_per_sec":0,"num_objects_recovered":196,"num_bytes_recovered":802816,"num_keys_recovered":0,"write_bytes_sec":1527106,"read_op_per_sec":0,"write_op_per_sec":372},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:38.819 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:38.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:39.149 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 193 20 135811 135791 2.74753 1.62109 0.0788368 0.0284261 2024-09-18T11:57:39.362 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:39.381 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:39.395 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-18T11:57:39.089792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":553193543,"num_objects":135076,"num_object_clones":0,"num_object_copies":270152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2691,"num_objects_unfound":0,"num_objects_dirty":135076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119131,"num_write_kb":476398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4335,"num_bytes_recovered":17662588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135139,"ondisk_log_size":135139,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2002276,"kb_used_data":1096268,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281113244,"statfs":{"total":289910292480,"available":287859961856,"internally_reserved":0,"allocated":1122578432,"data_stored":1121830883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,6,14,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21884928,"num_objects":5343,"num_object_clones":0,"num_object_copies":10686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-215,"num_objects_unfound":0,"num_objects_dirty":5343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5343,"num_write_kb":21372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":880640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002856"},"pg_stats":[{"pgid":"2.f","version":"39'8456","reported_seq":8527,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:37.033190+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:37.033190+0000","last_peered":"2024-09-18T11:57:37.033190+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:37.033190+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34635776,"num_objects":8456,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8456,"num_write_kb":33824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4857},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8457","reported_seq":8615,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:37.743823+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:37.743823+0000","last_peered":"2024-09-18T11:57:37.743823+0000","last_clean":"2024-09-18T11:57:37.743823+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:37.743823+0000","last_undegraded":"2024-09-18T11:57:37.743823+0000","last_fullsized":"2024-09-18T11:57:37.743823+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8457,"log_dups_size":0,"ondisk_log_size":8457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34615296,"num_objects":8451,"num_object_clones":0,"num_object_copies":16902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5649,"num_write_kb":22596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8537","reported_seq":8718,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:37.771567+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:37.771567+0000","last_peered":"2024-09-18T11:57:37.771567+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:37.771567+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8537,"log_dups_size":0,"ondisk_log_size":8537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34951168,"num_objects":8533,"num_object_clones":0,"num_object_copies":17066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5395,"num_write_kb":21580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":4997}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8387","reported_seq":8416,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:37.795714+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:37.795714+0000","last_peered":"2024-09-18T11:57:37.795714+0000","last_clean":"2024-09-18T11:57:37.795714+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:37.795714+0000","last_undegraded":"2024-09-18T11:57:37.795714+0000","last_fullsized":"2024-09-18T11:57:37.795714+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8387,"log_dups_size":0,"ondisk_log_size":8387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34353152,"num_objects":8387,"num_object_clones":0,"num_object_copies":16774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8387,"num_write_kb":33548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8474","reported_seq":8665,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:37.071189+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:37.071189+0000","last_peered":"2024-09-18T11:57:37.071189+0000","last_clean":"2024-09-18T11:57:37.071189+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:37.071189+0000","last_undegraded":"2024-09-18T11:57:37.071189+0000","last_fullsized":"2024-09-18T11:57:37.071189+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8474,"log_dups_size":0,"ondisk_log_size":8474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34680832,"num_objects":8467,"num_object_clones":0,"num_object_copies":16934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":22556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8422","reported_seq":8503,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:37.052060+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:37.052060+0000","last_peered":"2024-09-18T11:57:37.052060+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:37.052060+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8422,"log_dups_size":0,"ondisk_log_size":8422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34477266,"num_objects":8419,"num_object_clones":0,"num_object_copies":16838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8420,"num_write_kb":33671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4892},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8391","reported_seq":8478,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:37.115679+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:37.115679+0000","last_peered":"2024-09-18T11:57:37.115679+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:37.115679+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34346194,"num_objects":8387,"num_object_clones":0,"num_object_copies":16774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":33544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4828},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8546","reported_seq":8771,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:37.046031+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:37.046031+0000","last_peered":"2024-09-18T11:57:37.046031+0000","last_clean":"2024-09-18T11:57:37.046031+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:37.046031+0000","last_undegraded":"2024-09-18T11:57:37.046031+0000","last_fullsized":"2024-09-18T11:57:37.046031+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34996224,"num_objects":8544,"num_object_clones":0,"num_object_copies":17088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4863,"num_write_kb":19452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8395","reported_seq":15523,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:37.167727+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:37.167727+0000","last_peered":"2024-09-18T11:57:37.167727+0000","last_clean":"2024-09-18T11:57:37.167727+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:37.167727+0000","last_undegraded":"2024-09-18T11:57:37.167727+0000","last_fullsized":"2024-09-18T11:57:37.167727+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8395,"log_dups_size":0,"ondisk_log_size":8395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34358482,"num_objects":8390,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8393,"num_write_kb":33556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8418","reported_seq":8496,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:36.888829+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:36.888829+0000","last_peered":"2024-09-18T11:57:36.888829+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:36.888829+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8418,"log_dups_size":0,"ondisk_log_size":8418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34448594,"num_objects":8412,"num_object_clones":0,"num_object_copies":16824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8416,"num_write_kb":33644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4880},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8459","reported_seq":8515,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:37.699792+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:37.699792+0000","last_peered":"2024-09-18T11:57:37.699792+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:37.699792+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8459,"log_dups_size":0,"ondisk_log_size":8459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34628818,"num_objects":8456,"num_object_clones":0,"num_object_copies":16912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8457,"num_write_kb":33819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":4906}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8477","reported_seq":8686,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:37.770990+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:37.770990+0000","last_peered":"2024-09-18T11:57:37.770990+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:37.770990+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8477,"log_dups_size":0,"ondisk_log_size":8477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34697216,"num_objects":8471,"num_object_clones":0,"num_object_copies":16942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4957,"num_write_kb":19828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":4878}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8403","reported_seq":8485,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:37.109226+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:37.109226+0000","last_peered":"2024-09-18T11:57:37.109226+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:37.109226+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34411113,"num_objects":8402,"num_object_clones":0,"num_object_copies":16804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8402,"num_write_kb":33605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4867},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8483","reported_seq":10229,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:37.092454+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:37.092454+0000","last_peered":"2024-09-18T11:57:37.092454+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:37.092454+0000","last_undegraded":"2024-09-18T11:57:37.092454+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8483,"log_dups_size":0,"ondisk_log_size":8483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34715451,"num_objects":8478,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2691,"num_objects_unfound":0,"num_objects_dirty":8478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8480,"num_write_kb":33904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":834,"num_bytes_recovered":3403776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5787},{"shards":"0,2","objects":2691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8340","reported_seq":8420,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:37.099079+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:37.099079+0000","last_peered":"2024-09-18T11:57:37.099079+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:37.099079+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8340,"log_dups_size":0,"ondisk_log_size":8340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34145490,"num_objects":8338,"num_object_clones":0,"num_object_copies":16676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8338,"num_write_kb":33346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4745},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8491","reported_seq":8576,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:37.019776+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:37.019776+0000","last_peered":"2024-09-18T11:57:37.019776+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:37.019776+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8491,"log_dups_size":0,"ondisk_log_size":8491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34732452,"num_objects":8483,"num_object_clones":0,"num_object_copies":16966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":33923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4960},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":553193524,"num_objects":135074,"num_object_clones":0,"num_object_copies":270148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2691,"num_objects_unfound":0,"num_objects_dirty":135074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119129,"num_write_kb":476396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4334,"num_bytes_recovered":17662569,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1124155392,"data_stored":1124075375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135136,"ondisk_log_size":135136,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051629,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690300,"kb_used_data":384116,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93681540,"statfs":{"total":96636764160,"available":95929896960,"internally_reserved":0,"allocated":393334784,"data_stored":393111209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":3.6030000000000002},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":10.795999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":7.9850000000000003},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":3.569}]}]},{"osd":1,"up_from":35,"seq":150323855392,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553640,"kb_used_data":251616,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93818200,"statfs":{"total":96636764160,"available":96069836800,"internally_reserved":0,"allocated":257654784,"data_stored":257358007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.35699999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.30199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247856,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758336,"kb_used_data":460536,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93613504,"statfs":{"total":96636764160,"available":95860228096,"internally_reserved":0,"allocated":471588864,"data_stored":471361667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.42299999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":5.2169999999999996},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":472391680,"data_stored":472388201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257982464,"data_stored":257909405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":393781248,"data_stored":393777769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:40.149 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 194 20 136237 136217 2.74195 1.66406 0.0245825 0.028484 2024-09-18T11:57:41.149 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 195 20 136681 136661 2.73678 1.73438 0.0318619 0.0285374 2024-09-18T11:57:42.149 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 196 20 137137 137117 2.73191 1.78125 0.00861087 0.0285901 2024-09-18T11:57:42.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:43.128 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:43.131 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:43.149 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 197 20 137571 137551 2.72665 1.69531 0.0154052 0.0286441 2024-09-18T11:57:43.172 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-18T11:57:43.090723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562122823,"num_objects":137256,"num_object_clones":0,"num_object_copies":274512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2590,"num_objects_unfound":0,"num_objects_dirty":137256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121311,"num_write_kb":485118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4436,"num_bytes_recovered":18076284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137319,"ondisk_log_size":137319,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2036352,"kb_used_data":1113704,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281079168,"statfs":{"total":289910292480,"available":287825068032,"internally_reserved":0,"allocated":1140432896,"data_stored":1139685347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,10,5,6,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27381760,"num_objects":6685,"num_object_clones":0,"num_object_copies":13370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-316,"num_objects_unfound":0,"num_objects_dirty":6685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":26740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":1294336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002835"},"pg_stats":[{"pgid":"2.f","version":"39'8611","reported_seq":8682,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.130201+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:42.130201+0000","last_peered":"2024-09-18T11:57:42.130201+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:42.130201+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35270656,"num_objects":8611,"num_object_clones":0,"num_object_copies":17222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8611,"num_write_kb":34444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5012},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8594","reported_seq":8752,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.760919+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:42.760919+0000","last_peered":"2024-09-18T11:57:42.760919+0000","last_clean":"2024-09-18T11:57:42.760919+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:42.760919+0000","last_undegraded":"2024-09-18T11:57:42.760919+0000","last_fullsized":"2024-09-18T11:57:42.760919+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8594,"log_dups_size":0,"ondisk_log_size":8594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35176448,"num_objects":8588,"num_object_clones":0,"num_object_copies":17176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5786,"num_write_kb":23144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8681","reported_seq":8866,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.783811+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:42.783811+0000","last_peered":"2024-09-18T11:57:42.783811+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:42.783811+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8681,"log_dups_size":0,"ondisk_log_size":8681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35540992,"num_objects":8677,"num_object_clones":0,"num_object_copies":17354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5539,"num_write_kb":22156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5141}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8515","reported_seq":8544,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.777010+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:42.777010+0000","last_peered":"2024-09-18T11:57:42.777010+0000","last_clean":"2024-09-18T11:57:42.777010+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:42.777010+0000","last_undegraded":"2024-09-18T11:57:42.777010+0000","last_fullsized":"2024-09-18T11:57:42.777010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8515,"log_dups_size":0,"ondisk_log_size":8515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34877440,"num_objects":8515,"num_object_clones":0,"num_object_copies":17030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8515,"num_write_kb":34060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8589","reported_seq":8780,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.073670+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:42.073670+0000","last_peered":"2024-09-18T11:57:42.073670+0000","last_clean":"2024-09-18T11:57:42.073670+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:42.073670+0000","last_undegraded":"2024-09-18T11:57:42.073670+0000","last_fullsized":"2024-09-18T11:57:42.073670+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35151872,"num_objects":8582,"num_object_clones":0,"num_object_copies":17164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":23016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8546","reported_seq":8625,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.093094+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:42.093094+0000","last_peered":"2024-09-18T11:57:42.093094+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:42.093094+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34985170,"num_objects":8543,"num_object_clones":0,"num_object_copies":17086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8544,"num_write_kb":34167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5016},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8525","reported_seq":8612,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.079346+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:42.079346+0000","last_peered":"2024-09-18T11:57:42.079346+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:42.079346+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8525,"log_dups_size":0,"ondisk_log_size":8525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34895058,"num_objects":8521,"num_object_clones":0,"num_object_copies":17042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8523,"num_write_kb":34080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4962},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8665","reported_seq":8890,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.152693+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:42.152693+0000","last_peered":"2024-09-18T11:57:42.152693+0000","last_clean":"2024-09-18T11:57:42.152693+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:42.152693+0000","last_undegraded":"2024-09-18T11:57:42.152693+0000","last_fullsized":"2024-09-18T11:57:42.152693+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8665,"log_dups_size":0,"ondisk_log_size":8665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35483648,"num_objects":8663,"num_object_clones":0,"num_object_copies":17326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4982,"num_write_kb":19928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8534","reported_seq":15662,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.261312+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:42.261312+0000","last_peered":"2024-09-18T11:57:42.261312+0000","last_clean":"2024-09-18T11:57:42.261312+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:42.261312+0000","last_undegraded":"2024-09-18T11:57:42.261312+0000","last_fullsized":"2024-09-18T11:57:42.261312+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8534,"log_dups_size":0,"ondisk_log_size":8534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34927826,"num_objects":8529,"num_object_clones":0,"num_object_copies":17058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8532,"num_write_kb":34112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8559","reported_seq":8635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.061137+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:42.061137+0000","last_peered":"2024-09-18T11:57:42.061137+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:42.061137+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8559,"log_dups_size":0,"ondisk_log_size":8559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35026130,"num_objects":8553,"num_object_clones":0,"num_object_copies":17106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8557,"num_write_kb":34208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5021},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8579","reported_seq":8635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.823254+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:42.823254+0000","last_peered":"2024-09-18T11:57:42.823254+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:42.823254+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35120338,"num_objects":8576,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":34299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5026}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8612","reported_seq":8821,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.764376+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:42.764376+0000","last_peered":"2024-09-18T11:57:42.764376+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:42.764376+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35250176,"num_objects":8606,"num_object_clones":0,"num_object_copies":17212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":20368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5013}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8548","reported_seq":8630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.111846+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:42.111846+0000","last_peered":"2024-09-18T11:57:42.111846+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:42.111846+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8548,"log_dups_size":0,"ondisk_log_size":8548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35005033,"num_objects":8547,"num_object_clones":0,"num_object_copies":17094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8547,"num_write_kb":34185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5012},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8637","reported_seq":10584,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:42.174279+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:42.174279+0000","last_peered":"2024-09-18T11:57:42.174279+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:42.174279+0000","last_undegraded":"2024-09-18T11:57:42.174279+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8637,"log_dups_size":0,"ondisk_log_size":8637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35346235,"num_objects":8632,"num_object_clones":0,"num_object_copies":17264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2590,"num_objects_unfound":0,"num_objects_dirty":8632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8634,"num_write_kb":34520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":935,"num_bytes_recovered":3817472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6042},{"shards":"0,2","objects":2590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8490","reported_seq":8571,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.136274+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:42.136274+0000","last_peered":"2024-09-18T11:57:42.136274+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:42.136274+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8490,"log_dups_size":0,"ondisk_log_size":8490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34759890,"num_objects":8488,"num_object_clones":0,"num_object_copies":16976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":33946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4895},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8631","reported_seq":8715,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.172707+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:42.172707+0000","last_peered":"2024-09-18T11:57:42.172707+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:42.172707+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8631,"log_dups_size":0,"ondisk_log_size":8631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35305892,"num_objects":8623,"num_object_clones":0,"num_object_copies":17246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8628,"num_write_kb":34483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5100},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":562122804,"num_objects":137254,"num_object_clones":0,"num_object_copies":274508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2590,"num_objects_unfound":0,"num_objects_dirty":137254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121309,"num_write_kb":485116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4435,"num_bytes_recovered":18076265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1142439936,"data_stored":1142359919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137316,"ondisk_log_size":137316,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051630,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698572,"kb_used_data":388228,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93673268,"statfs":{"total":96636764160,"available":95921426432,"internally_reserved":0,"allocated":397545472,"data_stored":397321897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.20699999999999999},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.26900000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.23499999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.25}]}]},{"osd":1,"up_from":35,"seq":150323855393,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564712,"kb_used_data":258528,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93807128,"statfs":{"total":96636764160,"available":96058499072,"internally_reserved":0,"allocated":264732672,"data_stored":264435895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.189},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.59699999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.221}]}]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773068,"kb_used_data":466948,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93598772,"statfs":{"total":96636764160,"available":95845142528,"internally_reserved":0,"allocated":478154752,"data_stored":477927555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,1,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.214}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.5}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":479096832,"data_stored":479093353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265347072,"data_stored":265274013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397996032,"data_stored":397992553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:43.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:43.943 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:43.966 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:43.980 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-18T11:57:43.090723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562122823,"num_objects":137256,"num_object_clones":0,"num_object_copies":274512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2590,"num_objects_unfound":0,"num_objects_dirty":137256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121311,"num_write_kb":485118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4436,"num_bytes_recovered":18076284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137319,"ondisk_log_size":137319,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2036352,"kb_used_data":1113704,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281079168,"statfs":{"total":289910292480,"available":287825068032,"internally_reserved":0,"allocated":1140432896,"data_stored":1139685347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,10,5,6,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27381760,"num_objects":6685,"num_object_clones":0,"num_object_copies":13370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-316,"num_objects_unfound":0,"num_objects_dirty":6685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":26740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":1294336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002835"},"pg_stats":[{"pgid":"2.f","version":"39'8611","reported_seq":8682,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.130201+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:42.130201+0000","last_peered":"2024-09-18T11:57:42.130201+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:42.130201+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35270656,"num_objects":8611,"num_object_clones":0,"num_object_copies":17222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8611,"num_write_kb":34444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5012},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8594","reported_seq":8752,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.760919+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:42.760919+0000","last_peered":"2024-09-18T11:57:42.760919+0000","last_clean":"2024-09-18T11:57:42.760919+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:42.760919+0000","last_undegraded":"2024-09-18T11:57:42.760919+0000","last_fullsized":"2024-09-18T11:57:42.760919+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8594,"log_dups_size":0,"ondisk_log_size":8594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35176448,"num_objects":8588,"num_object_clones":0,"num_object_copies":17176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5786,"num_write_kb":23144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8681","reported_seq":8866,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.783811+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:42.783811+0000","last_peered":"2024-09-18T11:57:42.783811+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:42.783811+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8681,"log_dups_size":0,"ondisk_log_size":8681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35540992,"num_objects":8677,"num_object_clones":0,"num_object_copies":17354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5539,"num_write_kb":22156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5141}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8515","reported_seq":8544,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.777010+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:42.777010+0000","last_peered":"2024-09-18T11:57:42.777010+0000","last_clean":"2024-09-18T11:57:42.777010+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:42.777010+0000","last_undegraded":"2024-09-18T11:57:42.777010+0000","last_fullsized":"2024-09-18T11:57:42.777010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8515,"log_dups_size":0,"ondisk_log_size":8515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34877440,"num_objects":8515,"num_object_clones":0,"num_object_copies":17030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8515,"num_write_kb":34060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8589","reported_seq":8780,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.073670+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:42.073670+0000","last_peered":"2024-09-18T11:57:42.073670+0000","last_clean":"2024-09-18T11:57:42.073670+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:42.073670+0000","last_undegraded":"2024-09-18T11:57:42.073670+0000","last_fullsized":"2024-09-18T11:57:42.073670+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35151872,"num_objects":8582,"num_object_clones":0,"num_object_copies":17164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":23016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8546","reported_seq":8625,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.093094+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:42.093094+0000","last_peered":"2024-09-18T11:57:42.093094+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:42.093094+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34985170,"num_objects":8543,"num_object_clones":0,"num_object_copies":17086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8544,"num_write_kb":34167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5016},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8525","reported_seq":8612,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.079346+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:42.079346+0000","last_peered":"2024-09-18T11:57:42.079346+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:42.079346+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8525,"log_dups_size":0,"ondisk_log_size":8525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34895058,"num_objects":8521,"num_object_clones":0,"num_object_copies":17042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8523,"num_write_kb":34080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4962},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8665","reported_seq":8890,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.152693+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:42.152693+0000","last_peered":"2024-09-18T11:57:42.152693+0000","last_clean":"2024-09-18T11:57:42.152693+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:42.152693+0000","last_undegraded":"2024-09-18T11:57:42.152693+0000","last_fullsized":"2024-09-18T11:57:42.152693+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8665,"log_dups_size":0,"ondisk_log_size":8665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35483648,"num_objects":8663,"num_object_clones":0,"num_object_copies":17326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4982,"num_write_kb":19928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8534","reported_seq":15662,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.261312+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:42.261312+0000","last_peered":"2024-09-18T11:57:42.261312+0000","last_clean":"2024-09-18T11:57:42.261312+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:42.261312+0000","last_undegraded":"2024-09-18T11:57:42.261312+0000","last_fullsized":"2024-09-18T11:57:42.261312+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8534,"log_dups_size":0,"ondisk_log_size":8534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34927826,"num_objects":8529,"num_object_clones":0,"num_object_copies":17058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8532,"num_write_kb":34112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8559","reported_seq":8635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.061137+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:42.061137+0000","last_peered":"2024-09-18T11:57:42.061137+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:42.061137+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8559,"log_dups_size":0,"ondisk_log_size":8559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35026130,"num_objects":8553,"num_object_clones":0,"num_object_copies":17106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8557,"num_write_kb":34208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5021},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8579","reported_seq":8635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.823254+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:42.823254+0000","last_peered":"2024-09-18T11:57:42.823254+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:42.823254+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35120338,"num_objects":8576,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":34299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5026}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8612","reported_seq":8821,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.764376+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:42.764376+0000","last_peered":"2024-09-18T11:57:42.764376+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:42.764376+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35250176,"num_objects":8606,"num_object_clones":0,"num_object_copies":17212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":20368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5013}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8548","reported_seq":8630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.111846+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:42.111846+0000","last_peered":"2024-09-18T11:57:42.111846+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:42.111846+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8548,"log_dups_size":0,"ondisk_log_size":8548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35005033,"num_objects":8547,"num_object_clones":0,"num_object_copies":17094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8547,"num_write_kb":34185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5012},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8637","reported_seq":10584,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:42.174279+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:42.174279+0000","last_peered":"2024-09-18T11:57:42.174279+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:42.174279+0000","last_undegraded":"2024-09-18T11:57:42.174279+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8637,"log_dups_size":0,"ondisk_log_size":8637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35346235,"num_objects":8632,"num_object_clones":0,"num_object_copies":17264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2590,"num_objects_unfound":0,"num_objects_dirty":8632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8634,"num_write_kb":34520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":935,"num_bytes_recovered":3817472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6042},{"shards":"0,2","objects":2590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8490","reported_seq":8571,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.136274+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:42.136274+0000","last_peered":"2024-09-18T11:57:42.136274+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:42.136274+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8490,"log_dups_size":0,"ondisk_log_size":8490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34759890,"num_objects":8488,"num_object_clones":0,"num_object_copies":16976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":33946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4895},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8631","reported_seq":8715,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.172707+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:42.172707+0000","last_peered":"2024-09-18T11:57:42.172707+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:42.172707+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8631,"log_dups_size":0,"ondisk_log_size":8631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35305892,"num_objects":8623,"num_object_clones":0,"num_object_copies":17246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8628,"num_write_kb":34483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5100},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":562122804,"num_objects":137254,"num_object_clones":0,"num_object_copies":274508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2590,"num_objects_unfound":0,"num_objects_dirty":137254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121309,"num_write_kb":485116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4435,"num_bytes_recovered":18076265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1142439936,"data_stored":1142359919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137316,"ondisk_log_size":137316,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051630,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698572,"kb_used_data":388228,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93673268,"statfs":{"total":96636764160,"available":95921426432,"internally_reserved":0,"allocated":397545472,"data_stored":397321897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.20699999999999999},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.26900000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.23499999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.25}]}]},{"osd":1,"up_from":35,"seq":150323855393,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564712,"kb_used_data":258528,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93807128,"statfs":{"total":96636764160,"available":96058499072,"internally_reserved":0,"allocated":264732672,"data_stored":264435895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.189},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.59699999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.221}]}]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773068,"kb_used_data":466948,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93598772,"statfs":{"total":96636764160,"available":95845142528,"internally_reserved":0,"allocated":478154752,"data_stored":477927555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,1,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.214}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.5}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":479096832,"data_stored":479093353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265347072,"data_stored":265274013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397996032,"data_stored":397992553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:43.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:57:43.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'8611", 'reported_seq': 8682, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:42.130201+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:57:42.130201+0000', 'last_peered': '2024-09-18T11:57:42.130201+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:57:42.130201+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8611, 'log_dups_size': 0, 'ondisk_log_size': 8611, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35270656, 'num_objects': 8611, 'num_object_clones': 0, 'num_object_copies': 17222, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8611, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8611, 'num_write_kb': 34444, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5012}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:43.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:57:43.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'8681", 'reported_seq': 8866, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:42.783811+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:57:42.783811+0000', 'last_peered': '2024-09-18T11:57:42.783811+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:57:42.783811+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 8681, 'log_dups_size': 0, 'ondisk_log_size': 8681, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35540992, 'num_objects': 8677, 'num_object_clones': 0, 'num_object_copies': 17354, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5539, 'num_write_kb': 22156, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 5141}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:43.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:57:43.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'8546", 'reported_seq': 8625, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:42.093094+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:57:42.093094+0000', 'last_peered': '2024-09-18T11:57:42.093094+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:57:42.093094+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8546, 'log_dups_size': 0, 'ondisk_log_size': 8546, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34985170, 'num_objects': 8543, 'num_object_clones': 0, 'num_object_copies': 17086, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8544, 'num_write_kb': 34167, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5016}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:43.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:57:43.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'8525", 'reported_seq': 8612, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:42.079346+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:57:42.079346+0000', 'last_peered': '2024-09-18T11:57:42.079346+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:57:42.079346+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8525, 'log_dups_size': 0, 'ondisk_log_size': 8525, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34895058, 'num_objects': 8521, 'num_object_clones': 0, 'num_object_copies': 17042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8523, 'num_write_kb': 34080, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4962}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:43.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:57:43.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'8559", 'reported_seq': 8635, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:42.061137+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:57:42.061137+0000', 'last_peered': '2024-09-18T11:57:42.061137+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:57:42.061137+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8559, 'log_dups_size': 0, 'ondisk_log_size': 8559, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35026130, 'num_objects': 8553, 'num_object_clones': 0, 'num_object_copies': 17106, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8557, 'num_write_kb': 34208, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5021}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:43.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:57:43.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'8579", 'reported_seq': 8635, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:42.823254+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:57:42.823254+0000', 'last_peered': '2024-09-18T11:57:42.823254+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:57:42.823254+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8579, 'log_dups_size': 0, 'ondisk_log_size': 8579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35120338, 'num_objects': 8576, 'num_object_clones': 0, 'num_object_copies': 17152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8577, 'num_write_kb': 34299, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 5026}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:43.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:57:43.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'8612", 'reported_seq': 8821, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:42.764376+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:57:42.764376+0000', 'last_peered': '2024-09-18T11:57:42.764376+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:57:42.764376+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 8612, 'log_dups_size': 0, 'ondisk_log_size': 8612, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35250176, 'num_objects': 8606, 'num_object_clones': 0, 'num_object_copies': 17212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5092, 'num_write_kb': 20368, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 5013}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:43.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:57:43.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'8548", 'reported_seq': 8630, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:42.111846+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:57:42.111846+0000', 'last_peered': '2024-09-18T11:57:42.111846+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:57:42.111846+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8548, 'log_dups_size': 0, 'ondisk_log_size': 8548, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35005033, 'num_objects': 8547, 'num_object_clones': 0, 'num_object_copies': 17094, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8547, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8547, 'num_write_kb': 34185, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5012}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:43.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:57:43.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'8637", 'reported_seq': 10584, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:57:42.174279+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:57:42.174279+0000', 'last_peered': '2024-09-18T11:57:42.174279+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:57:42.174279+0000', 'last_undegraded': '2024-09-18T11:57:42.174279+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8637, 'log_dups_size': 0, 'ondisk_log_size': 8637, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35346235, 'num_objects': 8632, 'num_object_clones': 0, 'num_object_copies': 17264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2590, 'num_objects_unfound': 0, 'num_objects_dirty': 8632, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8634, 'num_write_kb': 34520, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 935, 'num_bytes_recovered': 3817472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6042}, {'shards': '0,2', 'objects': 2590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:43.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:57:43.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'8490", 'reported_seq': 8571, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:42.136274+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:57:42.136274+0000', 'last_peered': '2024-09-18T11:57:42.136274+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:57:42.136274+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8490, 'log_dups_size': 0, 'ondisk_log_size': 8490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34759890, 'num_objects': 8488, 'num_object_clones': 0, 'num_object_copies': 16976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8488, 'num_write_kb': 33946, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4895}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:43.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:57:43.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'8631", 'reported_seq': 8715, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:42.172707+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:57:42.172707+0000', 'last_peered': '2024-09-18T11:57:42.172707+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:57:42.172707+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8631, 'log_dups_size': 0, 'ondisk_log_size': 8631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35305892, 'num_objects': 8623, 'num_object_clones': 0, 'num_object_copies': 17246, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8623, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8628, 'num_write_kb': 34483, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5100}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:43.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:44.149 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 198 20 137997 137977 2.72128 1.66406 0.0968174 0.0286975 2024-09-18T11:57:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:44.772 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/274512 objects degraded (12.955%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":264,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":137256,"data_bytes":562122823,"bytes_used":2085224448,"bytes_avail":287825068032,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":274512,"degraded_ratio":0.12955353500029143,"misplaced_objects":2590,"misplaced_total":274512,"misplaced_ratio":0.0094349245206038351,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":107835,"recovering_keys_per_sec":0,"num_objects_recovered":316,"num_bytes_recovered":1294336,"num_keys_recovered":0,"write_bytes_sec":2281274,"read_op_per_sec":0,"write_op_per_sec":556},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:44.822 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:44.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:45.150 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 199 20 138411 138391 2.71573 1.61719 0.0401512 0.0287578 2024-09-18T11:57:45.516 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:45.526 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:45.551 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-18T11:57:45.092338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562122823,"num_objects":137256,"num_object_clones":0,"num_object_copies":274512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2590,"num_objects_unfound":0,"num_objects_dirty":137256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121311,"num_write_kb":485118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4436,"num_bytes_recovered":18076284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137319,"ondisk_log_size":137319,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2036352,"kb_used_data":1113704,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281079168,"statfs":{"total":289910292480,"available":287825068032,"internally_reserved":0,"allocated":1140432896,"data_stored":1139685347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,10,5,6,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18153472,"num_objects":4432,"num_object_clones":0,"num_object_copies":8864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4432,"num_write_kb":17728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":868352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004006"},"pg_stats":[{"pgid":"2.f","version":"39'8611","reported_seq":8682,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.130201+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:42.130201+0000","last_peered":"2024-09-18T11:57:42.130201+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:42.130201+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8611,"log_dups_size":0,"ondisk_log_size":8611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35270656,"num_objects":8611,"num_object_clones":0,"num_object_copies":17222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8611,"num_write_kb":34444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5012},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8594","reported_seq":8752,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.760919+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:42.760919+0000","last_peered":"2024-09-18T11:57:42.760919+0000","last_clean":"2024-09-18T11:57:42.760919+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:42.760919+0000","last_undegraded":"2024-09-18T11:57:42.760919+0000","last_fullsized":"2024-09-18T11:57:42.760919+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8594,"log_dups_size":0,"ondisk_log_size":8594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35176448,"num_objects":8588,"num_object_clones":0,"num_object_copies":17176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5786,"num_write_kb":23144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8681","reported_seq":8866,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.783811+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:42.783811+0000","last_peered":"2024-09-18T11:57:42.783811+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:42.783811+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8681,"log_dups_size":0,"ondisk_log_size":8681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35540992,"num_objects":8677,"num_object_clones":0,"num_object_copies":17354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5539,"num_write_kb":22156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5141}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8515","reported_seq":8544,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.777010+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:42.777010+0000","last_peered":"2024-09-18T11:57:42.777010+0000","last_clean":"2024-09-18T11:57:42.777010+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:42.777010+0000","last_undegraded":"2024-09-18T11:57:42.777010+0000","last_fullsized":"2024-09-18T11:57:42.777010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8515,"log_dups_size":0,"ondisk_log_size":8515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34877440,"num_objects":8515,"num_object_clones":0,"num_object_copies":17030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8515,"num_write_kb":34060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8589","reported_seq":8780,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.073670+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:42.073670+0000","last_peered":"2024-09-18T11:57:42.073670+0000","last_clean":"2024-09-18T11:57:42.073670+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:42.073670+0000","last_undegraded":"2024-09-18T11:57:42.073670+0000","last_fullsized":"2024-09-18T11:57:42.073670+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35151872,"num_objects":8582,"num_object_clones":0,"num_object_copies":17164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":23016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8546","reported_seq":8625,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.093094+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:42.093094+0000","last_peered":"2024-09-18T11:57:42.093094+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:42.093094+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8546,"log_dups_size":0,"ondisk_log_size":8546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34985170,"num_objects":8543,"num_object_clones":0,"num_object_copies":17086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8544,"num_write_kb":34167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5016},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8525","reported_seq":8612,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.079346+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:42.079346+0000","last_peered":"2024-09-18T11:57:42.079346+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:42.079346+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8525,"log_dups_size":0,"ondisk_log_size":8525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34895058,"num_objects":8521,"num_object_clones":0,"num_object_copies":17042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8523,"num_write_kb":34080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4962},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8665","reported_seq":8890,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.152693+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:42.152693+0000","last_peered":"2024-09-18T11:57:42.152693+0000","last_clean":"2024-09-18T11:57:42.152693+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:42.152693+0000","last_undegraded":"2024-09-18T11:57:42.152693+0000","last_fullsized":"2024-09-18T11:57:42.152693+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8665,"log_dups_size":0,"ondisk_log_size":8665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35483648,"num_objects":8663,"num_object_clones":0,"num_object_copies":17326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4982,"num_write_kb":19928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8534","reported_seq":15662,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:42.261312+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:42.261312+0000","last_peered":"2024-09-18T11:57:42.261312+0000","last_clean":"2024-09-18T11:57:42.261312+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:42.261312+0000","last_undegraded":"2024-09-18T11:57:42.261312+0000","last_fullsized":"2024-09-18T11:57:42.261312+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8534,"log_dups_size":0,"ondisk_log_size":8534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34927826,"num_objects":8529,"num_object_clones":0,"num_object_copies":17058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8532,"num_write_kb":34112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8559","reported_seq":8635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.061137+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:42.061137+0000","last_peered":"2024-09-18T11:57:42.061137+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:42.061137+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8559,"log_dups_size":0,"ondisk_log_size":8559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35026130,"num_objects":8553,"num_object_clones":0,"num_object_copies":17106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8557,"num_write_kb":34208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5021},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8579","reported_seq":8635,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.823254+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:42.823254+0000","last_peered":"2024-09-18T11:57:42.823254+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:42.823254+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35120338,"num_objects":8576,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":34299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5026}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8612","reported_seq":8821,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.764376+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:42.764376+0000","last_peered":"2024-09-18T11:57:42.764376+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:42.764376+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35250176,"num_objects":8606,"num_object_clones":0,"num_object_copies":17212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":20368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5013}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8548","reported_seq":8630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.111846+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:42.111846+0000","last_peered":"2024-09-18T11:57:42.111846+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:42.111846+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8548,"log_dups_size":0,"ondisk_log_size":8548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35005033,"num_objects":8547,"num_object_clones":0,"num_object_copies":17094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8547,"num_write_kb":34185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5012},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8637","reported_seq":10584,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:42.174279+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:42.174279+0000","last_peered":"2024-09-18T11:57:42.174279+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:42.174279+0000","last_undegraded":"2024-09-18T11:57:42.174279+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8637,"log_dups_size":0,"ondisk_log_size":8637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35346235,"num_objects":8632,"num_object_clones":0,"num_object_copies":17264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2590,"num_objects_unfound":0,"num_objects_dirty":8632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8634,"num_write_kb":34520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":935,"num_bytes_recovered":3817472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6042},{"shards":"0,2","objects":2590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8490","reported_seq":8571,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.136274+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:42.136274+0000","last_peered":"2024-09-18T11:57:42.136274+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:42.136274+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8490,"log_dups_size":0,"ondisk_log_size":8490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34759890,"num_objects":8488,"num_object_clones":0,"num_object_copies":16976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":33946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4895},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8631","reported_seq":8715,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:42.172707+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:42.172707+0000","last_peered":"2024-09-18T11:57:42.172707+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:42.172707+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8631,"log_dups_size":0,"ondisk_log_size":8631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35305892,"num_objects":8623,"num_object_clones":0,"num_object_copies":17246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8628,"num_write_kb":34483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5100},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":562122804,"num_objects":137254,"num_object_clones":0,"num_object_copies":274508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2590,"num_objects_unfound":0,"num_objects_dirty":137254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121309,"num_write_kb":485116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4435,"num_bytes_recovered":18076265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1142439936,"data_stored":1142359919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137316,"ondisk_log_size":137316,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051630,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698572,"kb_used_data":388228,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93673268,"statfs":{"total":96636764160,"available":95921426432,"internally_reserved":0,"allocated":397545472,"data_stored":397321897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.20699999999999999},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.26900000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.23499999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.25}]}]},{"osd":1,"up_from":35,"seq":150323855393,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564712,"kb_used_data":258528,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93807128,"statfs":{"total":96636764160,"available":96058499072,"internally_reserved":0,"allocated":264732672,"data_stored":264435895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.189},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.59699999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.221}]}]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":773068,"kb_used_data":466948,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93598772,"statfs":{"total":96636764160,"available":95845142528,"internally_reserved":0,"allocated":478154752,"data_stored":477927555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,1,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.214}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.5}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":479096832,"data_stored":479093353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265347072,"data_stored":265274013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397996032,"data_stored":397992553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:46.150 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:2024-09-18T11:57:46.150904+0000 min lat: 0.0021114 max lat: 2.57663 avg lat: 0.0288116 2024-09-18T11:57:46.150 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:57:46.150 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 200 20 138857 138837 2.71086 1.74219 0.0608051 0.0288116 2024-09-18T11:57:47.150 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 201 20 139296 139276 2.70591 1.71484 0.0869178 0.0288633 2024-09-18T11:57:48.150 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 202 20 139751 139731 2.70131 1.77734 0.106188 0.0289123 2024-09-18T11:57:48.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:49.150 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 203 20 140171 140151 2.69609 1.64062 0.0564451 0.0289687 2024-09-18T11:57:49.251 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:49.286 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:49.301 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-18T11:57:49.093218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":570978375,"num_objects":139418,"num_object_clones":0,"num_object_copies":278836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2492,"num_objects_unfound":0,"num_objects_dirty":139418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123473,"num_write_kb":493766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4534,"num_bytes_recovered":18477692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139481,"ondisk_log_size":139481,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2050176,"kb_used_data":1131688,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281065344,"statfs":{"total":289910292480,"available":287810912256,"internally_reserved":0,"allocated":1158848512,"data_stored":1158100963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,7,6,8,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27009024,"num_objects":6594,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":6594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6594,"num_write_kb":26376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003900"},"pg_stats":[{"pgid":"2.f","version":"39'8749","reported_seq":8815,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.152248+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:47.152248+0000","last_peered":"2024-09-18T11:57:47.152248+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:47.152248+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8749,"log_dups_size":0,"ondisk_log_size":8749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35835904,"num_objects":8749,"num_object_clones":0,"num_object_copies":17498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8749,"num_write_kb":34996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5150},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8734","reported_seq":8891,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.836779+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:47.836779+0000","last_peered":"2024-09-18T11:57:47.836779+0000","last_clean":"2024-09-18T11:57:47.836779+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:47.836779+0000","last_undegraded":"2024-09-18T11:57:47.836779+0000","last_fullsized":"2024-09-18T11:57:47.836779+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8734,"log_dups_size":0,"ondisk_log_size":8734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35749888,"num_objects":8728,"num_object_clones":0,"num_object_copies":17456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":23704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8792","reported_seq":8977,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.781759+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:47.781759+0000","last_peered":"2024-09-18T11:57:47.781759+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:47.781759+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35995648,"num_objects":8788,"num_object_clones":0,"num_object_copies":17576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5650,"num_write_kb":22600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8654","reported_seq":8683,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.782264+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:47.782264+0000","last_peered":"2024-09-18T11:57:47.782264+0000","last_clean":"2024-09-18T11:57:47.782264+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:47.782264+0000","last_undegraded":"2024-09-18T11:57:47.782264+0000","last_fullsized":"2024-09-18T11:57:47.782264+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8654,"log_dups_size":0,"ondisk_log_size":8654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35446784,"num_objects":8654,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":34616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8728","reported_seq":8919,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.130919+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:47.130919+0000","last_peered":"2024-09-18T11:57:47.130919+0000","last_clean":"2024-09-18T11:57:47.130919+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:47.130919+0000","last_undegraded":"2024-09-18T11:57:47.130919+0000","last_fullsized":"2024-09-18T11:57:47.130919+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35721216,"num_objects":8721,"num_object_clones":0,"num_object_copies":17442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5893,"num_write_kb":23572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8676","reported_seq":8757,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.079863+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:47.079863+0000","last_peered":"2024-09-18T11:57:47.079863+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:47.079863+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8676,"log_dups_size":0,"ondisk_log_size":8676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35517650,"num_objects":8673,"num_object_clones":0,"num_object_copies":17346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8674,"num_write_kb":34687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5146},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8678","reported_seq":8765,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.149504+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:47.149504+0000","last_peered":"2024-09-18T11:57:47.149504+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:47.149504+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8678,"log_dups_size":0,"ondisk_log_size":8678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35521746,"num_objects":8674,"num_object_clones":0,"num_object_copies":17348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8676,"num_write_kb":34692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5115},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8805","reported_seq":9029,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.157378+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:47.157378+0000","last_peered":"2024-09-18T11:57:47.157378+0000","last_clean":"2024-09-18T11:57:47.157378+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:47.157378+0000","last_undegraded":"2024-09-18T11:57:47.157378+0000","last_fullsized":"2024-09-18T11:57:47.157378+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8805,"log_dups_size":0,"ondisk_log_size":8805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36057088,"num_objects":8803,"num_object_clones":0,"num_object_copies":17606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8653","reported_seq":15781,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.354677+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:47.354677+0000","last_peered":"2024-09-18T11:57:47.354677+0000","last_clean":"2024-09-18T11:57:47.354677+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:47.354677+0000","last_undegraded":"2024-09-18T11:57:47.354677+0000","last_fullsized":"2024-09-18T11:57:47.354677+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8653,"log_dups_size":0,"ondisk_log_size":8653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35415250,"num_objects":8648,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8651,"num_write_kb":34588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8697","reported_seq":8774,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.131119+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:47.131119+0000","last_peered":"2024-09-18T11:57:47.131119+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:47.131119+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8697,"log_dups_size":0,"ondisk_log_size":8697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35591378,"num_objects":8691,"num_object_clones":0,"num_object_copies":17382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8695,"num_write_kb":34760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5159},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8709","reported_seq":8764,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.697384+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:47.697384+0000","last_peered":"2024-09-18T11:57:47.697384+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:47.697384+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35652818,"num_objects":8706,"num_object_clones":0,"num_object_copies":17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8707,"num_write_kb":34819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5156}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8755","reported_seq":8964,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.779919+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:47.779919+0000","last_peered":"2024-09-18T11:57:47.779919+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:47.779919+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8755,"log_dups_size":0,"ondisk_log_size":8755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35835904,"num_objects":8749,"num_object_clones":0,"num_object_copies":17498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5235,"num_write_kb":20940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5156}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8686","reported_seq":8768,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.105812+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:47.105812+0000","last_peered":"2024-09-18T11:57:47.105812+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:47.105812+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8686,"log_dups_size":0,"ondisk_log_size":8686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35570281,"num_objects":8685,"num_object_clones":0,"num_object_copies":17370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8685,"num_write_kb":34737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5150},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8775","reported_seq":10919,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:47.114979+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:47.114979+0000","last_peered":"2024-09-18T11:57:47.114979+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:47.114979+0000","last_undegraded":"2024-09-18T11:57:47.114979+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8775,"log_dups_size":0,"ondisk_log_size":8775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35911483,"num_objects":8770,"num_object_clones":0,"num_object_copies":17540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2492,"num_objects_unfound":0,"num_objects_dirty":8770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8772,"num_write_kb":35072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1033,"num_bytes_recovered":4218880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6278},{"shards":"0,2","objects":2492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8623","reported_seq":8704,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.134600+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:47.134600+0000","last_peered":"2024-09-18T11:57:47.134600+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:47.134600+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8623,"log_dups_size":0,"ondisk_log_size":8623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35304658,"num_objects":8621,"num_object_clones":0,"num_object_copies":17242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8621,"num_write_kb":34478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5028},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8764","reported_seq":8849,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.163089+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:47.163089+0000","last_peered":"2024-09-18T11:57:47.163089+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:47.163089+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35850660,"num_objects":8756,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8761,"num_write_kb":35015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5233},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":570978356,"num_objects":139416,"num_object_clones":0,"num_object_copies":278832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2492,"num_objects_unfound":0,"num_objects_dirty":139416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123471,"num_write_kb":493764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":18477673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1160650752,"data_stored":1160570735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139478,"ondisk_log_size":139478,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051631,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690360,"kb_used_data":392496,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93681480,"statfs":{"total":96636764160,"available":95929835520,"internally_reserved":0,"allocated":401915904,"data_stored":401692329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,0,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.318},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.27100000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.35799999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855394,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575944,"kb_used_data":265600,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93795896,"statfs":{"total":96636764160,"available":96046997504,"internally_reserved":0,"allocated":271974400,"data_stored":271677623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.17399999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":11.864000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":11.996}]}]},{"osd":0,"up_from":23,"seq":98784247858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":783872,"kb_used_data":473592,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93587968,"statfs":{"total":96636764160,"available":95834079232,"internally_reserved":0,"allocated":484958208,"data_stored":484731011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.22800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":485908480,"data_stored":485905001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272379904,"data_stored":272306845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402362368,"data_stored":402358889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:49.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:49.933 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:49.948 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:49.966 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-18T11:57:49.093218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":570978375,"num_objects":139418,"num_object_clones":0,"num_object_copies":278836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2492,"num_objects_unfound":0,"num_objects_dirty":139418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123473,"num_write_kb":493766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4534,"num_bytes_recovered":18477692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139481,"ondisk_log_size":139481,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2050176,"kb_used_data":1131688,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281065344,"statfs":{"total":289910292480,"available":287810912256,"internally_reserved":0,"allocated":1158848512,"data_stored":1158100963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,7,6,8,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27009024,"num_objects":6594,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":6594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6594,"num_write_kb":26376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003900"},"pg_stats":[{"pgid":"2.f","version":"39'8749","reported_seq":8815,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.152248+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:47.152248+0000","last_peered":"2024-09-18T11:57:47.152248+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:47.152248+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8749,"log_dups_size":0,"ondisk_log_size":8749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35835904,"num_objects":8749,"num_object_clones":0,"num_object_copies":17498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8749,"num_write_kb":34996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5150},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8734","reported_seq":8891,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.836779+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:47.836779+0000","last_peered":"2024-09-18T11:57:47.836779+0000","last_clean":"2024-09-18T11:57:47.836779+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:47.836779+0000","last_undegraded":"2024-09-18T11:57:47.836779+0000","last_fullsized":"2024-09-18T11:57:47.836779+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8734,"log_dups_size":0,"ondisk_log_size":8734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35749888,"num_objects":8728,"num_object_clones":0,"num_object_copies":17456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":23704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8792","reported_seq":8977,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.781759+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:47.781759+0000","last_peered":"2024-09-18T11:57:47.781759+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:47.781759+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35995648,"num_objects":8788,"num_object_clones":0,"num_object_copies":17576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5650,"num_write_kb":22600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8654","reported_seq":8683,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.782264+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:47.782264+0000","last_peered":"2024-09-18T11:57:47.782264+0000","last_clean":"2024-09-18T11:57:47.782264+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:47.782264+0000","last_undegraded":"2024-09-18T11:57:47.782264+0000","last_fullsized":"2024-09-18T11:57:47.782264+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8654,"log_dups_size":0,"ondisk_log_size":8654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35446784,"num_objects":8654,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":34616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8728","reported_seq":8919,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.130919+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:47.130919+0000","last_peered":"2024-09-18T11:57:47.130919+0000","last_clean":"2024-09-18T11:57:47.130919+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:47.130919+0000","last_undegraded":"2024-09-18T11:57:47.130919+0000","last_fullsized":"2024-09-18T11:57:47.130919+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35721216,"num_objects":8721,"num_object_clones":0,"num_object_copies":17442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5893,"num_write_kb":23572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8676","reported_seq":8757,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.079863+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:47.079863+0000","last_peered":"2024-09-18T11:57:47.079863+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:47.079863+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8676,"log_dups_size":0,"ondisk_log_size":8676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35517650,"num_objects":8673,"num_object_clones":0,"num_object_copies":17346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8674,"num_write_kb":34687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5146},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8678","reported_seq":8765,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.149504+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:47.149504+0000","last_peered":"2024-09-18T11:57:47.149504+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:47.149504+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8678,"log_dups_size":0,"ondisk_log_size":8678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35521746,"num_objects":8674,"num_object_clones":0,"num_object_copies":17348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8676,"num_write_kb":34692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5115},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8805","reported_seq":9029,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.157378+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:47.157378+0000","last_peered":"2024-09-18T11:57:47.157378+0000","last_clean":"2024-09-18T11:57:47.157378+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:47.157378+0000","last_undegraded":"2024-09-18T11:57:47.157378+0000","last_fullsized":"2024-09-18T11:57:47.157378+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8805,"log_dups_size":0,"ondisk_log_size":8805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36057088,"num_objects":8803,"num_object_clones":0,"num_object_copies":17606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8653","reported_seq":15781,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.354677+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:47.354677+0000","last_peered":"2024-09-18T11:57:47.354677+0000","last_clean":"2024-09-18T11:57:47.354677+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:47.354677+0000","last_undegraded":"2024-09-18T11:57:47.354677+0000","last_fullsized":"2024-09-18T11:57:47.354677+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8653,"log_dups_size":0,"ondisk_log_size":8653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35415250,"num_objects":8648,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8651,"num_write_kb":34588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8697","reported_seq":8774,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.131119+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:47.131119+0000","last_peered":"2024-09-18T11:57:47.131119+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:47.131119+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8697,"log_dups_size":0,"ondisk_log_size":8697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35591378,"num_objects":8691,"num_object_clones":0,"num_object_copies":17382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8695,"num_write_kb":34760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5159},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8709","reported_seq":8764,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.697384+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:47.697384+0000","last_peered":"2024-09-18T11:57:47.697384+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:47.697384+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35652818,"num_objects":8706,"num_object_clones":0,"num_object_copies":17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8707,"num_write_kb":34819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5156}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8755","reported_seq":8964,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.779919+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:47.779919+0000","last_peered":"2024-09-18T11:57:47.779919+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:47.779919+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8755,"log_dups_size":0,"ondisk_log_size":8755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35835904,"num_objects":8749,"num_object_clones":0,"num_object_copies":17498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5235,"num_write_kb":20940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5156}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8686","reported_seq":8768,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.105812+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:47.105812+0000","last_peered":"2024-09-18T11:57:47.105812+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:47.105812+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8686,"log_dups_size":0,"ondisk_log_size":8686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35570281,"num_objects":8685,"num_object_clones":0,"num_object_copies":17370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8685,"num_write_kb":34737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5150},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8775","reported_seq":10919,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:47.114979+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:47.114979+0000","last_peered":"2024-09-18T11:57:47.114979+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:47.114979+0000","last_undegraded":"2024-09-18T11:57:47.114979+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8775,"log_dups_size":0,"ondisk_log_size":8775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35911483,"num_objects":8770,"num_object_clones":0,"num_object_copies":17540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2492,"num_objects_unfound":0,"num_objects_dirty":8770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8772,"num_write_kb":35072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1033,"num_bytes_recovered":4218880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6278},{"shards":"0,2","objects":2492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8623","reported_seq":8704,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.134600+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:47.134600+0000","last_peered":"2024-09-18T11:57:47.134600+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:47.134600+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8623,"log_dups_size":0,"ondisk_log_size":8623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35304658,"num_objects":8621,"num_object_clones":0,"num_object_copies":17242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8621,"num_write_kb":34478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5028},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8764","reported_seq":8849,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.163089+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:47.163089+0000","last_peered":"2024-09-18T11:57:47.163089+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:47.163089+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35850660,"num_objects":8756,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8761,"num_write_kb":35015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5233},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":570978356,"num_objects":139416,"num_object_clones":0,"num_object_copies":278832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2492,"num_objects_unfound":0,"num_objects_dirty":139416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123471,"num_write_kb":493764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":18477673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1160650752,"data_stored":1160570735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139478,"ondisk_log_size":139478,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051631,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690360,"kb_used_data":392496,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93681480,"statfs":{"total":96636764160,"available":95929835520,"internally_reserved":0,"allocated":401915904,"data_stored":401692329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,0,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.318},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.27100000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.35799999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855394,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575944,"kb_used_data":265600,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93795896,"statfs":{"total":96636764160,"available":96046997504,"internally_reserved":0,"allocated":271974400,"data_stored":271677623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.17399999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":11.864000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":11.996}]}]},{"osd":0,"up_from":23,"seq":98784247858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":783872,"kb_used_data":473592,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93587968,"statfs":{"total":96636764160,"available":95834079232,"internally_reserved":0,"allocated":484958208,"data_stored":484731011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.22800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":485908480,"data_stored":485905001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272379904,"data_stored":272306845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402362368,"data_stored":402358889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:49.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:57:49.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'8749", 'reported_seq': 8815, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:47.152248+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:57:47.152248+0000', 'last_peered': '2024-09-18T11:57:47.152248+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:57:47.152248+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8749, 'log_dups_size': 0, 'ondisk_log_size': 8749, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35835904, 'num_objects': 8749, 'num_object_clones': 0, 'num_object_copies': 17498, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8749, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8749, 'num_write_kb': 34996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5150}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:49.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:57:49.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'8792", 'reported_seq': 8977, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:47.781759+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:57:47.781759+0000', 'last_peered': '2024-09-18T11:57:47.781759+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:57:47.781759+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 8792, 'log_dups_size': 0, 'ondisk_log_size': 8792, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35995648, 'num_objects': 8788, 'num_object_clones': 0, 'num_object_copies': 17576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5650, 'num_write_kb': 22600, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 5252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:49.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:57:49.967 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'8676", 'reported_seq': 8757, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:47.079863+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:57:47.079863+0000', 'last_peered': '2024-09-18T11:57:47.079863+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:57:47.079863+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8676, 'log_dups_size': 0, 'ondisk_log_size': 8676, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35517650, 'num_objects': 8673, 'num_object_clones': 0, 'num_object_copies': 17346, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8673, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8674, 'num_write_kb': 34687, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5146}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:49.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:57:49.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'8678", 'reported_seq': 8765, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:47.149504+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:57:47.149504+0000', 'last_peered': '2024-09-18T11:57:47.149504+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:57:47.149504+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8678, 'log_dups_size': 0, 'ondisk_log_size': 8678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35521746, 'num_objects': 8674, 'num_object_clones': 0, 'num_object_copies': 17348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8674, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8676, 'num_write_kb': 34692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5115}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:49.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:57:49.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'8697", 'reported_seq': 8774, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:47.131119+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:57:47.131119+0000', 'last_peered': '2024-09-18T11:57:47.131119+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:57:47.131119+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8697, 'log_dups_size': 0, 'ondisk_log_size': 8697, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35591378, 'num_objects': 8691, 'num_object_clones': 0, 'num_object_copies': 17382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8695, 'num_write_kb': 34760, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5159}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:49.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:57:49.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'8709", 'reported_seq': 8764, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:47.697384+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:57:47.697384+0000', 'last_peered': '2024-09-18T11:57:47.697384+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:57:47.697384+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8709, 'log_dups_size': 0, 'ondisk_log_size': 8709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35652818, 'num_objects': 8706, 'num_object_clones': 0, 'num_object_copies': 17412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8707, 'num_write_kb': 34819, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 5156}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:49.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:57:49.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'8755", 'reported_seq': 8964, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:47.779919+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:57:47.779919+0000', 'last_peered': '2024-09-18T11:57:47.779919+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:57:47.779919+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 8755, 'log_dups_size': 0, 'ondisk_log_size': 8755, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35835904, 'num_objects': 8749, 'num_object_clones': 0, 'num_object_copies': 17498, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8749, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5235, 'num_write_kb': 20940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 5156}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:49.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:57:49.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'8686", 'reported_seq': 8768, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:47.105812+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:57:47.105812+0000', 'last_peered': '2024-09-18T11:57:47.105812+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:57:47.105812+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8686, 'log_dups_size': 0, 'ondisk_log_size': 8686, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35570281, 'num_objects': 8685, 'num_object_clones': 0, 'num_object_copies': 17370, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8685, 'num_write_kb': 34737, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5150}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:49.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:57:49.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'8775", 'reported_seq': 10919, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:57:47.114979+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:57:47.114979+0000', 'last_peered': '2024-09-18T11:57:47.114979+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:57:47.114979+0000', 'last_undegraded': '2024-09-18T11:57:47.114979+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8775, 'log_dups_size': 0, 'ondisk_log_size': 8775, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35911483, 'num_objects': 8770, 'num_object_clones': 0, 'num_object_copies': 17540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2492, 'num_objects_unfound': 0, 'num_objects_dirty': 8770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8772, 'num_write_kb': 35072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1033, 'num_bytes_recovered': 4218880, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6278}, {'shards': '0,2', 'objects': 2492}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:49.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:57:49.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'8623", 'reported_seq': 8704, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:47.134600+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:57:47.134600+0000', 'last_peered': '2024-09-18T11:57:47.134600+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:57:47.134600+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8623, 'log_dups_size': 0, 'ondisk_log_size': 8623, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35304658, 'num_objects': 8621, 'num_object_clones': 0, 'num_object_copies': 17242, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8621, 'num_write_kb': 34478, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5028}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:49.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:57:49.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'8764", 'reported_seq': 8849, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:47.163089+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:57:47.163089+0000', 'last_peered': '2024-09-18T11:57:47.163089+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:57:47.163089+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8764, 'log_dups_size': 0, 'ondisk_log_size': 8764, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35850660, 'num_objects': 8756, 'num_object_clones': 0, 'num_object_copies': 17512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8761, 'num_write_kb': 35015, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5233}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:49.969 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:50.150 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 204 20 140572 140552 2.69055 1.56641 0.0392666 0.0290243 2024-09-18T11:57:50.940 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:50.940 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/278836 objects degraded (12.754%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":270,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":139418,"data_bytes":570978375,"bytes_used":2099380224,"bytes_avail":287810912256,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":278836,"degraded_ratio":0.12754450644823481,"misplaced_objects":2492,"misplaced_total":278836,"misplaced_ratio":0.0089371530218479676,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":105778,"recovering_keys_per_sec":0,"num_objects_recovered":310,"num_bytes_recovered":1269760,"num_keys_recovered":0,"write_bytes_sec":2250020,"read_op_per_sec":0,"write_op_per_sec":549},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:50.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:50.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:51.150 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 205 20 140998 140978 2.68554 1.66406 0.0666798 0.0290817 2024-09-18T11:57:51.708 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:51.721 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:51.742 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-18T11:57:51.093748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":570978375,"num_objects":139418,"num_object_clones":0,"num_object_copies":278836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2492,"num_objects_unfound":0,"num_objects_dirty":139418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123473,"num_write_kb":493766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4534,"num_bytes_recovered":18477692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139481,"ondisk_log_size":139481,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2050176,"kb_used_data":1131688,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281065344,"statfs":{"total":289910292480,"available":287810912256,"internally_reserved":0,"allocated":1158848512,"data_stored":1158100963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,7,6,8,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17784832,"num_objects":4342,"num_object_clones":0,"num_object_copies":8684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-199,"num_objects_unfound":0,"num_objects_dirty":4342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4342,"num_write_kb":17368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":815104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003955"},"pg_stats":[{"pgid":"2.f","version":"39'8749","reported_seq":8815,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.152248+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:47.152248+0000","last_peered":"2024-09-18T11:57:47.152248+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:47.152248+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8749,"log_dups_size":0,"ondisk_log_size":8749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35835904,"num_objects":8749,"num_object_clones":0,"num_object_copies":17498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8749,"num_write_kb":34996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5150},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8734","reported_seq":8891,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.836779+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:47.836779+0000","last_peered":"2024-09-18T11:57:47.836779+0000","last_clean":"2024-09-18T11:57:47.836779+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:47.836779+0000","last_undegraded":"2024-09-18T11:57:47.836779+0000","last_fullsized":"2024-09-18T11:57:47.836779+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8734,"log_dups_size":0,"ondisk_log_size":8734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35749888,"num_objects":8728,"num_object_clones":0,"num_object_copies":17456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":23704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8792","reported_seq":8977,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.781759+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:47.781759+0000","last_peered":"2024-09-18T11:57:47.781759+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:47.781759+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35995648,"num_objects":8788,"num_object_clones":0,"num_object_copies":17576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5650,"num_write_kb":22600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8654","reported_seq":8683,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.782264+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:47.782264+0000","last_peered":"2024-09-18T11:57:47.782264+0000","last_clean":"2024-09-18T11:57:47.782264+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:47.782264+0000","last_undegraded":"2024-09-18T11:57:47.782264+0000","last_fullsized":"2024-09-18T11:57:47.782264+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8654,"log_dups_size":0,"ondisk_log_size":8654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35446784,"num_objects":8654,"num_object_clones":0,"num_object_copies":17308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":34616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8728","reported_seq":8919,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.130919+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:47.130919+0000","last_peered":"2024-09-18T11:57:47.130919+0000","last_clean":"2024-09-18T11:57:47.130919+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:47.130919+0000","last_undegraded":"2024-09-18T11:57:47.130919+0000","last_fullsized":"2024-09-18T11:57:47.130919+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35721216,"num_objects":8721,"num_object_clones":0,"num_object_copies":17442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5893,"num_write_kb":23572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8676","reported_seq":8757,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.079863+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:47.079863+0000","last_peered":"2024-09-18T11:57:47.079863+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:47.079863+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8676,"log_dups_size":0,"ondisk_log_size":8676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35517650,"num_objects":8673,"num_object_clones":0,"num_object_copies":17346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8674,"num_write_kb":34687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5146},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8678","reported_seq":8765,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.149504+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:47.149504+0000","last_peered":"2024-09-18T11:57:47.149504+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:47.149504+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8678,"log_dups_size":0,"ondisk_log_size":8678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35521746,"num_objects":8674,"num_object_clones":0,"num_object_copies":17348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8676,"num_write_kb":34692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5115},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8805","reported_seq":9029,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.157378+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:47.157378+0000","last_peered":"2024-09-18T11:57:47.157378+0000","last_clean":"2024-09-18T11:57:47.157378+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:47.157378+0000","last_undegraded":"2024-09-18T11:57:47.157378+0000","last_fullsized":"2024-09-18T11:57:47.157378+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8805,"log_dups_size":0,"ondisk_log_size":8805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36057088,"num_objects":8803,"num_object_clones":0,"num_object_copies":17606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8653","reported_seq":15781,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:47.354677+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:47.354677+0000","last_peered":"2024-09-18T11:57:47.354677+0000","last_clean":"2024-09-18T11:57:47.354677+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:47.354677+0000","last_undegraded":"2024-09-18T11:57:47.354677+0000","last_fullsized":"2024-09-18T11:57:47.354677+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8653,"log_dups_size":0,"ondisk_log_size":8653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35415250,"num_objects":8648,"num_object_clones":0,"num_object_copies":17296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8651,"num_write_kb":34588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8697","reported_seq":8774,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.131119+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:47.131119+0000","last_peered":"2024-09-18T11:57:47.131119+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:47.131119+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8697,"log_dups_size":0,"ondisk_log_size":8697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35591378,"num_objects":8691,"num_object_clones":0,"num_object_copies":17382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8695,"num_write_kb":34760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5159},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8709","reported_seq":8764,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.697384+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:47.697384+0000","last_peered":"2024-09-18T11:57:47.697384+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:47.697384+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35652818,"num_objects":8706,"num_object_clones":0,"num_object_copies":17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8707,"num_write_kb":34819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5156}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8755","reported_seq":8964,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.779919+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:47.779919+0000","last_peered":"2024-09-18T11:57:47.779919+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:47.779919+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8755,"log_dups_size":0,"ondisk_log_size":8755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35835904,"num_objects":8749,"num_object_clones":0,"num_object_copies":17498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5235,"num_write_kb":20940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5156}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8686","reported_seq":8768,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.105812+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:47.105812+0000","last_peered":"2024-09-18T11:57:47.105812+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:47.105812+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8686,"log_dups_size":0,"ondisk_log_size":8686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35570281,"num_objects":8685,"num_object_clones":0,"num_object_copies":17370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8685,"num_write_kb":34737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5150},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8775","reported_seq":10919,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:47.114979+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:47.114979+0000","last_peered":"2024-09-18T11:57:47.114979+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:47.114979+0000","last_undegraded":"2024-09-18T11:57:47.114979+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8775,"log_dups_size":0,"ondisk_log_size":8775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35911483,"num_objects":8770,"num_object_clones":0,"num_object_copies":17540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2492,"num_objects_unfound":0,"num_objects_dirty":8770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8772,"num_write_kb":35072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1033,"num_bytes_recovered":4218880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6278},{"shards":"0,2","objects":2492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8623","reported_seq":8704,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.134600+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:47.134600+0000","last_peered":"2024-09-18T11:57:47.134600+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:47.134600+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8623,"log_dups_size":0,"ondisk_log_size":8623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35304658,"num_objects":8621,"num_object_clones":0,"num_object_copies":17242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8621,"num_write_kb":34478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5028},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8764","reported_seq":8849,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:47.163089+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:47.163089+0000","last_peered":"2024-09-18T11:57:47.163089+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:47.163089+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35850660,"num_objects":8756,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8761,"num_write_kb":35015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5233},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":570978356,"num_objects":139416,"num_object_clones":0,"num_object_copies":278832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2492,"num_objects_unfound":0,"num_objects_dirty":139416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123471,"num_write_kb":493764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4533,"num_bytes_recovered":18477673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1160650752,"data_stored":1160570735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139478,"ondisk_log_size":139478,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051631,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690360,"kb_used_data":392496,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93681480,"statfs":{"total":96636764160,"available":95929835520,"internally_reserved":0,"allocated":401915904,"data_stored":401692329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,0,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.318},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.27100000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.35799999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855394,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575944,"kb_used_data":265600,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93795896,"statfs":{"total":96636764160,"available":96046997504,"internally_reserved":0,"allocated":271974400,"data_stored":271677623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.17399999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":11.864000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":11.996}]}]},{"osd":0,"up_from":23,"seq":98784247858,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":783872,"kb_used_data":473592,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93587968,"statfs":{"total":96636764160,"available":95834079232,"internally_reserved":0,"allocated":484958208,"data_stored":484731011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.26900000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.22800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":485908480,"data_stored":485905001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":272379904,"data_stored":272306845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402362368,"data_stored":402358889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:52.154 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 206 20 141416 141396 2.68038 1.63281 0.0242191 0.0291382 2024-09-18T11:57:53.154 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 207 20 141863 141843 2.67587 1.74609 0.0353402 0.0291893 2024-09-18T11:57:54.154 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 208 20 142295 142275 2.67112 1.6875 0.103993 0.0292394 2024-09-18T11:57:54.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:55.154 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 209 20 142722 142702 2.66632 1.66797 0.0463833 0.0292902 2024-09-18T11:57:55.437 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:55.458 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:55.472 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-18T11:57:55.094736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":579665991,"num_objects":141539,"num_object_clones":0,"num_object_copies":283078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2384,"num_objects_unfound":0,"num_objects_dirty":141539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125594,"num_write_kb":502250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4642,"num_bytes_recovered":18920060,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141602,"ondisk_log_size":141602,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2051120,"kb_used_data":1149272,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281064400,"statfs":{"total":289910292480,"available":287809945600,"internally_reserved":0,"allocated":1176854528,"data_stored":1176106979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,2,5,6,10,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17543168,"num_objects":4283,"num_object_clones":0,"num_object_copies":8566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":4283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4283,"num_write_kb":17132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004012"},"pg_stats":[{"pgid":"2.f","version":"39'8901","reported_seq":8972,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.183263+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:52.183263+0000","last_peered":"2024-09-18T11:57:52.183263+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:52.183263+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36458496,"num_objects":8901,"num_object_clones":0,"num_object_copies":17802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8901,"num_write_kb":35604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5302},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8857","reported_seq":9014,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.777419+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:52.777419+0000","last_peered":"2024-09-18T11:57:52.777419+0000","last_clean":"2024-09-18T11:57:52.777419+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:52.777419+0000","last_undegraded":"2024-09-18T11:57:52.777419+0000","last_fullsized":"2024-09-18T11:57:52.777419+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36253696,"num_objects":8851,"num_object_clones":0,"num_object_copies":17702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":24196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8919","reported_seq":9104,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.763223+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:52.763223+0000","last_peered":"2024-09-18T11:57:52.763223+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:52.763223+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8919,"log_dups_size":0,"ondisk_log_size":8919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36515840,"num_objects":8915,"num_object_clones":0,"num_object_copies":17830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5777,"num_write_kb":23108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8768","reported_seq":8797,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.799399+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:52.799399+0000","last_peered":"2024-09-18T11:57:52.799399+0000","last_clean":"2024-09-18T11:57:52.799399+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:52.799399+0000","last_undegraded":"2024-09-18T11:57:52.799399+0000","last_fullsized":"2024-09-18T11:57:52.799399+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8768,"log_dups_size":0,"ondisk_log_size":8768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913728,"num_objects":8768,"num_object_clones":0,"num_object_copies":17536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8768,"num_write_kb":35072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8850","reported_seq":9041,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.179969+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:52.179969+0000","last_peered":"2024-09-18T11:57:52.179969+0000","last_clean":"2024-09-18T11:57:52.179969+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:52.179969+0000","last_undegraded":"2024-09-18T11:57:52.179969+0000","last_fullsized":"2024-09-18T11:57:52.179969+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8850,"log_dups_size":0,"ondisk_log_size":8850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36220928,"num_objects":8843,"num_object_clones":0,"num_object_copies":17686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":24060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8812","reported_seq":8893,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:51.925334+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:51.925334+0000","last_peered":"2024-09-18T11:57:51.925334+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:51.925334+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8812,"log_dups_size":0,"ondisk_log_size":8812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36074706,"num_objects":8809,"num_object_clones":0,"num_object_copies":17618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8810,"num_write_kb":35231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5282},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8805","reported_seq":8892,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.143531+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:52.143531+0000","last_peered":"2024-09-18T11:57:52.143531+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:52.143531+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8805,"log_dups_size":0,"ondisk_log_size":8805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36041938,"num_objects":8801,"num_object_clones":0,"num_object_copies":17602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8803,"num_write_kb":35200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5242},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8956","reported_seq":9180,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.195524+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:52.195524+0000","last_peered":"2024-09-18T11:57:52.195524+0000","last_clean":"2024-09-18T11:57:52.195524+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:52.195524+0000","last_undegraded":"2024-09-18T11:57:52.195524+0000","last_fullsized":"2024-09-18T11:57:52.195524+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8956,"log_dups_size":0,"ondisk_log_size":8956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36675584,"num_objects":8954,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5273,"num_write_kb":21092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8779","reported_seq":15907,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.445235+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:52.445235+0000","last_peered":"2024-09-18T11:57:52.445235+0000","last_clean":"2024-09-18T11:57:52.445235+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:52.445235+0000","last_undegraded":"2024-09-18T11:57:52.445235+0000","last_fullsized":"2024-09-18T11:57:52.445235+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8779,"log_dups_size":0,"ondisk_log_size":8779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35931346,"num_objects":8774,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":35092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8839","reported_seq":8917,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.203789+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:52.203789+0000","last_peered":"2024-09-18T11:57:52.203789+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:52.203789+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36173010,"num_objects":8833,"num_object_clones":0,"num_object_copies":17666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8837,"num_write_kb":35328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5301},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8840","reported_seq":8896,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.808780+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:52.808780+0000","last_peered":"2024-09-18T11:57:52.808780+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:52.808780+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8840,"log_dups_size":0,"ondisk_log_size":8840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36189394,"num_objects":8837,"num_object_clones":0,"num_object_copies":17674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8838,"num_write_kb":35343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8894","reported_seq":9103,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.646416+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:52.646416+0000","last_peered":"2024-09-18T11:57:52.646416+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:52.646416+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8894,"log_dups_size":0,"ondisk_log_size":8894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36405248,"num_objects":8888,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5374,"num_write_kb":21496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5295}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8833","reported_seq":8915,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.070903+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:52.070903+0000","last_peered":"2024-09-18T11:57:52.070903+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:52.070903+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8833,"log_dups_size":0,"ondisk_log_size":8833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36172393,"num_objects":8832,"num_object_clones":0,"num_object_copies":17664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8832,"num_write_kb":35325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5297},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8910","reported_seq":11271,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:52.209778+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:52.209778+0000","last_peered":"2024-09-18T11:57:52.209778+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:52.209778+0000","last_undegraded":"2024-09-18T11:57:52.209778+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36464443,"num_objects":8905,"num_object_clones":0,"num_object_copies":17810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2384,"num_objects_unfound":0,"num_objects_dirty":8905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8907,"num_write_kb":35612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":4661248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6521},{"shards":"0,2","objects":2384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8738","reported_seq":8819,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.014742+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:52.014742+0000","last_peered":"2024-09-18T11:57:52.014742+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:52.014742+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8738,"log_dups_size":0,"ondisk_log_size":8738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35775698,"num_objects":8736,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8736,"num_write_kb":34938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5143},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8898","reported_seq":8983,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.179876+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:52.179876+0000","last_peered":"2024-09-18T11:57:52.179876+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:52.179876+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8898,"log_dups_size":0,"ondisk_log_size":8898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36399524,"num_objects":8890,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8895,"num_write_kb":35551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5367},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":579665972,"num_objects":141537,"num_object_clones":0,"num_object_copies":283074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2384,"num_objects_unfound":0,"num_objects_dirty":141537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125592,"num_write_kb":502248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4641,"num_bytes_recovered":18920041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1178505216,"data_stored":1178425199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141599,"ondisk_log_size":141599,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051632,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698600,"kb_used_data":396576,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93673240,"statfs":{"total":96636764160,"available":95921397760,"internally_reserved":0,"allocated":406093824,"data_stored":405870249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.222},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.46300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.29199999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855395,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574532,"kb_used_data":272508,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93797308,"statfs":{"total":96636764160,"available":96048443392,"internally_reserved":0,"allocated":279048192,"data_stored":278751415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.251},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.30499999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":777988,"kb_used_data":480188,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93593852,"statfs":{"total":96636764160,"available":95840104448,"internally_reserved":0,"allocated":491712512,"data_stored":491485315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,1,6,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.252}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":492494848,"data_stored":492491369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279412736,"data_stored":279339677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":406597632,"data_stored":406594153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:55.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:56.155 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 210 20 143142 143122 2.66143 1.64062 0.0270927 0.0293448 2024-09-18T11:57:56.176 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:56.199 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:56.219 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-18T11:57:55.094736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":579665991,"num_objects":141539,"num_object_clones":0,"num_object_copies":283078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2384,"num_objects_unfound":0,"num_objects_dirty":141539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125594,"num_write_kb":502250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4642,"num_bytes_recovered":18920060,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141602,"ondisk_log_size":141602,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2051120,"kb_used_data":1149272,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281064400,"statfs":{"total":289910292480,"available":287809945600,"internally_reserved":0,"allocated":1176854528,"data_stored":1176106979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,2,5,6,10,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17543168,"num_objects":4283,"num_object_clones":0,"num_object_copies":8566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":4283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4283,"num_write_kb":17132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004012"},"pg_stats":[{"pgid":"2.f","version":"39'8901","reported_seq":8972,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.183263+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:52.183263+0000","last_peered":"2024-09-18T11:57:52.183263+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:52.183263+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36458496,"num_objects":8901,"num_object_clones":0,"num_object_copies":17802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8901,"num_write_kb":35604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5302},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8857","reported_seq":9014,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.777419+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:52.777419+0000","last_peered":"2024-09-18T11:57:52.777419+0000","last_clean":"2024-09-18T11:57:52.777419+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:52.777419+0000","last_undegraded":"2024-09-18T11:57:52.777419+0000","last_fullsized":"2024-09-18T11:57:52.777419+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36253696,"num_objects":8851,"num_object_clones":0,"num_object_copies":17702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":24196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8919","reported_seq":9104,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.763223+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:52.763223+0000","last_peered":"2024-09-18T11:57:52.763223+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:52.763223+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8919,"log_dups_size":0,"ondisk_log_size":8919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36515840,"num_objects":8915,"num_object_clones":0,"num_object_copies":17830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5777,"num_write_kb":23108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8768","reported_seq":8797,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.799399+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:52.799399+0000","last_peered":"2024-09-18T11:57:52.799399+0000","last_clean":"2024-09-18T11:57:52.799399+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:52.799399+0000","last_undegraded":"2024-09-18T11:57:52.799399+0000","last_fullsized":"2024-09-18T11:57:52.799399+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8768,"log_dups_size":0,"ondisk_log_size":8768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913728,"num_objects":8768,"num_object_clones":0,"num_object_copies":17536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8768,"num_write_kb":35072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8850","reported_seq":9041,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.179969+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:52.179969+0000","last_peered":"2024-09-18T11:57:52.179969+0000","last_clean":"2024-09-18T11:57:52.179969+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:52.179969+0000","last_undegraded":"2024-09-18T11:57:52.179969+0000","last_fullsized":"2024-09-18T11:57:52.179969+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8850,"log_dups_size":0,"ondisk_log_size":8850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36220928,"num_objects":8843,"num_object_clones":0,"num_object_copies":17686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":24060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8812","reported_seq":8893,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:51.925334+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:51.925334+0000","last_peered":"2024-09-18T11:57:51.925334+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:51.925334+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8812,"log_dups_size":0,"ondisk_log_size":8812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36074706,"num_objects":8809,"num_object_clones":0,"num_object_copies":17618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8810,"num_write_kb":35231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5282},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8805","reported_seq":8892,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.143531+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:52.143531+0000","last_peered":"2024-09-18T11:57:52.143531+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:52.143531+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8805,"log_dups_size":0,"ondisk_log_size":8805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36041938,"num_objects":8801,"num_object_clones":0,"num_object_copies":17602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8803,"num_write_kb":35200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5242},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8956","reported_seq":9180,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.195524+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:52.195524+0000","last_peered":"2024-09-18T11:57:52.195524+0000","last_clean":"2024-09-18T11:57:52.195524+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:52.195524+0000","last_undegraded":"2024-09-18T11:57:52.195524+0000","last_fullsized":"2024-09-18T11:57:52.195524+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8956,"log_dups_size":0,"ondisk_log_size":8956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36675584,"num_objects":8954,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5273,"num_write_kb":21092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8779","reported_seq":15907,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.445235+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:52.445235+0000","last_peered":"2024-09-18T11:57:52.445235+0000","last_clean":"2024-09-18T11:57:52.445235+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:52.445235+0000","last_undegraded":"2024-09-18T11:57:52.445235+0000","last_fullsized":"2024-09-18T11:57:52.445235+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8779,"log_dups_size":0,"ondisk_log_size":8779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35931346,"num_objects":8774,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":35092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8839","reported_seq":8917,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.203789+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:52.203789+0000","last_peered":"2024-09-18T11:57:52.203789+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:52.203789+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36173010,"num_objects":8833,"num_object_clones":0,"num_object_copies":17666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8837,"num_write_kb":35328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5301},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8840","reported_seq":8896,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.808780+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:52.808780+0000","last_peered":"2024-09-18T11:57:52.808780+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:52.808780+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8840,"log_dups_size":0,"ondisk_log_size":8840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36189394,"num_objects":8837,"num_object_clones":0,"num_object_copies":17674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8838,"num_write_kb":35343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8894","reported_seq":9103,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.646416+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:52.646416+0000","last_peered":"2024-09-18T11:57:52.646416+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:52.646416+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8894,"log_dups_size":0,"ondisk_log_size":8894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36405248,"num_objects":8888,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5374,"num_write_kb":21496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5295}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8833","reported_seq":8915,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.070903+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:52.070903+0000","last_peered":"2024-09-18T11:57:52.070903+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:52.070903+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8833,"log_dups_size":0,"ondisk_log_size":8833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36172393,"num_objects":8832,"num_object_clones":0,"num_object_copies":17664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8832,"num_write_kb":35325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5297},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8910","reported_seq":11271,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:52.209778+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:52.209778+0000","last_peered":"2024-09-18T11:57:52.209778+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:52.209778+0000","last_undegraded":"2024-09-18T11:57:52.209778+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36464443,"num_objects":8905,"num_object_clones":0,"num_object_copies":17810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2384,"num_objects_unfound":0,"num_objects_dirty":8905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8907,"num_write_kb":35612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":4661248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6521},{"shards":"0,2","objects":2384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8738","reported_seq":8819,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.014742+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:52.014742+0000","last_peered":"2024-09-18T11:57:52.014742+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:52.014742+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8738,"log_dups_size":0,"ondisk_log_size":8738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35775698,"num_objects":8736,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8736,"num_write_kb":34938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5143},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8898","reported_seq":8983,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.179876+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:52.179876+0000","last_peered":"2024-09-18T11:57:52.179876+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:52.179876+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8898,"log_dups_size":0,"ondisk_log_size":8898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36399524,"num_objects":8890,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8895,"num_write_kb":35551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5367},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":579665972,"num_objects":141537,"num_object_clones":0,"num_object_copies":283074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2384,"num_objects_unfound":0,"num_objects_dirty":141537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125592,"num_write_kb":502248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4641,"num_bytes_recovered":18920041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1178505216,"data_stored":1178425199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141599,"ondisk_log_size":141599,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051632,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698600,"kb_used_data":396576,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93673240,"statfs":{"total":96636764160,"available":95921397760,"internally_reserved":0,"allocated":406093824,"data_stored":405870249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.222},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.46300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.29199999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855395,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574532,"kb_used_data":272508,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93797308,"statfs":{"total":96636764160,"available":96048443392,"internally_reserved":0,"allocated":279048192,"data_stored":278751415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.251},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.30499999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":777988,"kb_used_data":480188,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93593852,"statfs":{"total":96636764160,"available":95840104448,"internally_reserved":0,"allocated":491712512,"data_stored":491485315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,1,6,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.252}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":492494848,"data_stored":492491369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279412736,"data_stored":279339677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":406597632,"data_stored":406594153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:56.220 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:57:56.220 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'8901", 'reported_seq': 8972, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:52.183263+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:57:52.183263+0000', 'last_peered': '2024-09-18T11:57:52.183263+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:57:52.183263+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8901, 'log_dups_size': 0, 'ondisk_log_size': 8901, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36458496, 'num_objects': 8901, 'num_object_clones': 0, 'num_object_copies': 17802, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8901, 'num_write_kb': 35604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5302}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:56.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:57:56.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'8919", 'reported_seq': 9104, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:52.763223+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:57:52.763223+0000', 'last_peered': '2024-09-18T11:57:52.763223+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:57:52.763223+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 8919, 'log_dups_size': 0, 'ondisk_log_size': 8919, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36515840, 'num_objects': 8915, 'num_object_clones': 0, 'num_object_copies': 17830, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8915, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5777, 'num_write_kb': 23108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 5379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:56.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:57:56.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'8812", 'reported_seq': 8893, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:51.925334+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:57:51.925334+0000', 'last_peered': '2024-09-18T11:57:51.925334+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:57:51.925334+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8812, 'log_dups_size': 0, 'ondisk_log_size': 8812, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36074706, 'num_objects': 8809, 'num_object_clones': 0, 'num_object_copies': 17618, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8809, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8810, 'num_write_kb': 35231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5282}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:56.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:57:56.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'8805", 'reported_seq': 8892, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:52.143531+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:57:52.143531+0000', 'last_peered': '2024-09-18T11:57:52.143531+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:57:52.143531+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8805, 'log_dups_size': 0, 'ondisk_log_size': 8805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36041938, 'num_objects': 8801, 'num_object_clones': 0, 'num_object_copies': 17602, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8803, 'num_write_kb': 35200, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5242}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:56.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:57:56.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'8839", 'reported_seq': 8917, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:52.203789+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:57:52.203789+0000', 'last_peered': '2024-09-18T11:57:52.203789+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:57:52.203789+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8839, 'log_dups_size': 0, 'ondisk_log_size': 8839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36173010, 'num_objects': 8833, 'num_object_clones': 0, 'num_object_copies': 17666, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8837, 'num_write_kb': 35328, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5301}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:56.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:57:56.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'8840", 'reported_seq': 8896, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:52.808780+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:57:52.808780+0000', 'last_peered': '2024-09-18T11:57:52.808780+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:57:52.808780+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8840, 'log_dups_size': 0, 'ondisk_log_size': 8840, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36189394, 'num_objects': 8837, 'num_object_clones': 0, 'num_object_copies': 17674, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8837, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8838, 'num_write_kb': 35343, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 5287}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:56.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:57:56.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'8894", 'reported_seq': 9103, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:52.646416+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:57:52.646416+0000', 'last_peered': '2024-09-18T11:57:52.646416+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:57:52.646416+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 8894, 'log_dups_size': 0, 'ondisk_log_size': 8894, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36405248, 'num_objects': 8888, 'num_object_clones': 0, 'num_object_copies': 17776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5374, 'num_write_kb': 21496, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 5295}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:57:56.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:57:56.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'8833", 'reported_seq': 8915, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:52.070903+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:57:52.070903+0000', 'last_peered': '2024-09-18T11:57:52.070903+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:57:52.070903+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8833, 'log_dups_size': 0, 'ondisk_log_size': 8833, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36172393, 'num_objects': 8832, 'num_object_clones': 0, 'num_object_copies': 17664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8832, 'num_write_kb': 35325, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5297}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:56.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:57:56.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'8910", 'reported_seq': 11271, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:57:52.209778+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:57:52.209778+0000', 'last_peered': '2024-09-18T11:57:52.209778+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:57:52.209778+0000', 'last_undegraded': '2024-09-18T11:57:52.209778+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8910, 'log_dups_size': 0, 'ondisk_log_size': 8910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36464443, 'num_objects': 8905, 'num_object_clones': 0, 'num_object_copies': 17810, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2384, 'num_objects_unfound': 0, 'num_objects_dirty': 8905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8907, 'num_write_kb': 35612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1141, 'num_bytes_recovered': 4661248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6521}, {'shards': '0,2', 'objects': 2384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:56.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:57:56.223 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'8738", 'reported_seq': 8819, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:52.014742+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:57:52.014742+0000', 'last_peered': '2024-09-18T11:57:52.014742+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:57:52.014742+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8738, 'log_dups_size': 0, 'ondisk_log_size': 8738, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35775698, 'num_objects': 8736, 'num_object_clones': 0, 'num_object_copies': 17472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8736, 'num_write_kb': 34938, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5143}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:56.223 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:57:56.223 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'8898", 'reported_seq': 8983, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:52.179876+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:57:52.179876+0000', 'last_peered': '2024-09-18T11:57:52.179876+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:57:52.179876+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8898, 'log_dups_size': 0, 'ondisk_log_size': 8898, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36399524, 'num_objects': 8890, 'num_object_clones': 0, 'num_object_copies': 17780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8895, 'num_write_kb': 35551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5367}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:57:56.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:57:57.158 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 211 20 143557 143537 2.65646 1.62109 0.0127891 0.0294011 2024-09-18T11:57:57.225 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:57.229 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/283078 objects degraded (12.563%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":277,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":141539,"data_bytes":579665991,"bytes_used":2100346880,"bytes_avail":287809945600,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":283078,"degraded_ratio":0.12563321769971528,"misplaced_objects":2384,"misplaced_total":283078,"misplaced_ratio":0.008421707091331717,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":70291,"recovering_keys_per_sec":0,"num_objects_recovered":206,"num_bytes_recovered":843776,"num_keys_recovered":0,"write_bytes_sec":1461442,"read_op_per_sec":0,"write_op_per_sec":356},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:57:57.264 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:57:57.264 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:57:57.887 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:57:57.898 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:57:57.924 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-18T11:57:57.095186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":579665991,"num_objects":141539,"num_object_clones":0,"num_object_copies":283078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2384,"num_objects_unfound":0,"num_objects_dirty":141539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125594,"num_write_kb":502250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4642,"num_bytes_recovered":18920060,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141602,"ondisk_log_size":141602,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2051120,"kb_used_data":1149272,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281064400,"statfs":{"total":289910292480,"available":287809945600,"internally_reserved":0,"allocated":1176854528,"data_stored":1176106979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,2,5,6,10,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17543168,"num_objects":4283,"num_object_clones":0,"num_object_copies":8566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":4283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4283,"num_write_kb":17132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002847"},"pg_stats":[{"pgid":"2.f","version":"39'8901","reported_seq":8972,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.183263+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:52.183263+0000","last_peered":"2024-09-18T11:57:52.183263+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:52.183263+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36458496,"num_objects":8901,"num_object_clones":0,"num_object_copies":17802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8901,"num_write_kb":35604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5302},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8857","reported_seq":9014,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.777419+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:52.777419+0000","last_peered":"2024-09-18T11:57:52.777419+0000","last_clean":"2024-09-18T11:57:52.777419+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:52.777419+0000","last_undegraded":"2024-09-18T11:57:52.777419+0000","last_fullsized":"2024-09-18T11:57:52.777419+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36253696,"num_objects":8851,"num_object_clones":0,"num_object_copies":17702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6049,"num_write_kb":24196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'8919","reported_seq":9104,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.763223+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:52.763223+0000","last_peered":"2024-09-18T11:57:52.763223+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:52.763223+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":8919,"log_dups_size":0,"ondisk_log_size":8919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36515840,"num_objects":8915,"num_object_clones":0,"num_object_copies":17830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5777,"num_write_kb":23108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8768","reported_seq":8797,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.799399+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:52.799399+0000","last_peered":"2024-09-18T11:57:52.799399+0000","last_clean":"2024-09-18T11:57:52.799399+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:52.799399+0000","last_undegraded":"2024-09-18T11:57:52.799399+0000","last_fullsized":"2024-09-18T11:57:52.799399+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8768,"log_dups_size":0,"ondisk_log_size":8768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913728,"num_objects":8768,"num_object_clones":0,"num_object_copies":17536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8768,"num_write_kb":35072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8850","reported_seq":9041,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.179969+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:52.179969+0000","last_peered":"2024-09-18T11:57:52.179969+0000","last_clean":"2024-09-18T11:57:52.179969+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:52.179969+0000","last_undegraded":"2024-09-18T11:57:52.179969+0000","last_fullsized":"2024-09-18T11:57:52.179969+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8850,"log_dups_size":0,"ondisk_log_size":8850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36220928,"num_objects":8843,"num_object_clones":0,"num_object_copies":17686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":24060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8812","reported_seq":8893,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:51.925334+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:51.925334+0000","last_peered":"2024-09-18T11:57:51.925334+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:51.925334+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8812,"log_dups_size":0,"ondisk_log_size":8812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36074706,"num_objects":8809,"num_object_clones":0,"num_object_copies":17618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8810,"num_write_kb":35231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5282},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8805","reported_seq":8892,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.143531+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:52.143531+0000","last_peered":"2024-09-18T11:57:52.143531+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:52.143531+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8805,"log_dups_size":0,"ondisk_log_size":8805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36041938,"num_objects":8801,"num_object_clones":0,"num_object_copies":17602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8803,"num_write_kb":35200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5242},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'8956","reported_seq":9180,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.195524+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:52.195524+0000","last_peered":"2024-09-18T11:57:52.195524+0000","last_clean":"2024-09-18T11:57:52.195524+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:52.195524+0000","last_undegraded":"2024-09-18T11:57:52.195524+0000","last_fullsized":"2024-09-18T11:57:52.195524+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":8956,"log_dups_size":0,"ondisk_log_size":8956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36675584,"num_objects":8954,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5273,"num_write_kb":21092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8779","reported_seq":15907,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:52.445235+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:52.445235+0000","last_peered":"2024-09-18T11:57:52.445235+0000","last_clean":"2024-09-18T11:57:52.445235+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:52.445235+0000","last_undegraded":"2024-09-18T11:57:52.445235+0000","last_fullsized":"2024-09-18T11:57:52.445235+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8779,"log_dups_size":0,"ondisk_log_size":8779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35931346,"num_objects":8774,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":35092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8839","reported_seq":8917,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.203789+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:52.203789+0000","last_peered":"2024-09-18T11:57:52.203789+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:52.203789+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36173010,"num_objects":8833,"num_object_clones":0,"num_object_copies":17666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8837,"num_write_kb":35328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5301},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8840","reported_seq":8896,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.808780+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:52.808780+0000","last_peered":"2024-09-18T11:57:52.808780+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:52.808780+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8840,"log_dups_size":0,"ondisk_log_size":8840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36189394,"num_objects":8837,"num_object_clones":0,"num_object_copies":17674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8838,"num_write_kb":35343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'8894","reported_seq":9103,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.646416+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:52.646416+0000","last_peered":"2024-09-18T11:57:52.646416+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:52.646416+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":8894,"log_dups_size":0,"ondisk_log_size":8894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36405248,"num_objects":8888,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5374,"num_write_kb":21496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5295}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8833","reported_seq":8915,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.070903+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:52.070903+0000","last_peered":"2024-09-18T11:57:52.070903+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:52.070903+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8833,"log_dups_size":0,"ondisk_log_size":8833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36172393,"num_objects":8832,"num_object_clones":0,"num_object_copies":17664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8832,"num_write_kb":35325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5297},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'8910","reported_seq":11271,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:52.209778+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:52.209778+0000","last_peered":"2024-09-18T11:57:52.209778+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:52.209778+0000","last_undegraded":"2024-09-18T11:57:52.209778+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36464443,"num_objects":8905,"num_object_clones":0,"num_object_copies":17810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2384,"num_objects_unfound":0,"num_objects_dirty":8905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8907,"num_write_kb":35612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":4661248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6521},{"shards":"0,2","objects":2384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8738","reported_seq":8819,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.014742+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:52.014742+0000","last_peered":"2024-09-18T11:57:52.014742+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:52.014742+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8738,"log_dups_size":0,"ondisk_log_size":8738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35775698,"num_objects":8736,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8736,"num_write_kb":34938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5143},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'8898","reported_seq":8983,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:52.179876+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:52.179876+0000","last_peered":"2024-09-18T11:57:52.179876+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:52.179876+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8898,"log_dups_size":0,"ondisk_log_size":8898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36399524,"num_objects":8890,"num_object_clones":0,"num_object_copies":17780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8895,"num_write_kb":35551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5367},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":579665972,"num_objects":141537,"num_object_clones":0,"num_object_copies":283074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2384,"num_objects_unfound":0,"num_objects_dirty":141537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125592,"num_write_kb":502248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4641,"num_bytes_recovered":18920041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1178505216,"data_stored":1178425199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141599,"ondisk_log_size":141599,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051632,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698600,"kb_used_data":396576,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93673240,"statfs":{"total":96636764160,"available":95921397760,"internally_reserved":0,"allocated":406093824,"data_stored":405870249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.222},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.46300000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.29199999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855395,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574532,"kb_used_data":272508,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93797308,"statfs":{"total":96636764160,"available":96048443392,"internally_reserved":0,"allocated":279048192,"data_stored":278751415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.251},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.30499999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":777988,"kb_used_data":480188,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93593852,"statfs":{"total":96636764160,"available":95840104448,"internally_reserved":0,"allocated":491712512,"data_stored":491485315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,1,6,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.252}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":492494848,"data_stored":492491369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":279412736,"data_stored":279339677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":406597632,"data_stored":406594153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:57:58.158 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 212 20 143968 143948 2.6515 1.60547 0.087169 0.0294573 2024-09-18T11:57:59.158 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 213 20 144408 144388 2.64712 1.71875 0.0183913 0.0295059 2024-09-18T11:58:00.160 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 214 20 144839 144819 2.64262 1.68359 0.01886 0.0295528 2024-09-18T11:58:00.927 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:01.158 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 215 20 145272 145252 2.6382 1.69141 0.0357029 0.0296024 2024-09-18T11:58:01.779 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:01.812 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:01.827 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-09-18T11:58:01.096139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":588476487,"num_objects":143690,"num_object_clones":0,"num_object_copies":287380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2262,"num_objects_unfound":0,"num_objects_dirty":143690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127745,"num_write_kb":510854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4764,"num_bytes_recovered":19419772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143753,"ondisk_log_size":143753,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2086152,"kb_used_data":1167664,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281029368,"statfs":{"total":289910292480,"available":287774072832,"internally_reserved":0,"allocated":1195687936,"data_stored":1194940387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,6,5,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17498112,"num_objects":4272,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-230,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":17088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002920"},"pg_stats":[{"pgid":"2.f","version":"39'9053","reported_seq":9124,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.237594+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:57.237594+0000","last_peered":"2024-09-18T11:57:57.237594+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:57.237594+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37081088,"num_objects":9053,"num_object_clones":0,"num_object_copies":18106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9053,"num_write_kb":36212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5454},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8990","reported_seq":9148,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:57.848907+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:57.848907+0000","last_peered":"2024-09-18T11:57:57.848907+0000","last_clean":"2024-09-18T11:57:57.848907+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:57.848907+0000","last_undegraded":"2024-09-18T11:57:57.848907+0000","last_fullsized":"2024-09-18T11:57:57.848907+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8990,"log_dups_size":0,"ondisk_log_size":8990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36798464,"num_objects":8984,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6182,"num_write_kb":24728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9036","reported_seq":9221,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.757696+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:57.757696+0000","last_peered":"2024-09-18T11:57:57.757696+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:57.757696+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9036,"log_dups_size":0,"ondisk_log_size":9036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36995072,"num_objects":9032,"num_object_clones":0,"num_object_copies":18064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5894,"num_write_kb":23576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5496}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8900","reported_seq":8929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:57.814320+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:57.814320+0000","last_peered":"2024-09-18T11:57:57.814320+0000","last_clean":"2024-09-18T11:57:57.814320+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:57.814320+0000","last_undegraded":"2024-09-18T11:57:57.814320+0000","last_fullsized":"2024-09-18T11:57:57.814320+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8900,"log_dups_size":0,"ondisk_log_size":8900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36454400,"num_objects":8900,"num_object_clones":0,"num_object_copies":17800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8900,"num_write_kb":35600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8976","reported_seq":9167,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:57.257823+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:57.257823+0000","last_peered":"2024-09-18T11:57:57.257823+0000","last_clean":"2024-09-18T11:57:57.257823+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:57.257823+0000","last_undegraded":"2024-09-18T11:57:57.257823+0000","last_fullsized":"2024-09-18T11:57:57.257823+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36737024,"num_objects":8969,"num_object_clones":0,"num_object_copies":17938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6141,"num_write_kb":24564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8944","reported_seq":9025,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.213556+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:57.213556+0000","last_peered":"2024-09-18T11:57:57.213556+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:57.213556+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8944,"log_dups_size":0,"ondisk_log_size":8944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36615378,"num_objects":8941,"num_object_clones":0,"num_object_copies":17882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8942,"num_write_kb":35759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5414},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8943","reported_seq":9030,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.113679+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:57.113679+0000","last_peered":"2024-09-18T11:57:57.113679+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:57.113679+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8943,"log_dups_size":0,"ondisk_log_size":8943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36607186,"num_objects":8939,"num_object_clones":0,"num_object_copies":17878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8941,"num_write_kb":35752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5380},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9093","reported_seq":9318,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:57.227239+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:57.227239+0000","last_peered":"2024-09-18T11:57:57.227239+0000","last_clean":"2024-09-18T11:57:57.227239+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:57.227239+0000","last_undegraded":"2024-09-18T11:57:57.227239+0000","last_fullsized":"2024-09-18T11:57:57.227239+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9093,"log_dups_size":0,"ondisk_log_size":9093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37236736,"num_objects":9091,"num_object_clones":0,"num_object_copies":18182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":21640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8920","reported_seq":16048,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:57.440046+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:57.440046+0000","last_peered":"2024-09-18T11:57:57.440046+0000","last_clean":"2024-09-18T11:57:57.440046+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:57.440046+0000","last_undegraded":"2024-09-18T11:57:57.440046+0000","last_fullsized":"2024-09-18T11:57:57.440046+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8920,"log_dups_size":0,"ondisk_log_size":8920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36508882,"num_objects":8915,"num_object_clones":0,"num_object_copies":17830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":35656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8987","reported_seq":9064,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.276627+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:57.276627+0000","last_peered":"2024-09-18T11:57:57.276627+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:57.276627+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8987,"log_dups_size":0,"ondisk_log_size":8987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36779218,"num_objects":8981,"num_object_clones":0,"num_object_copies":17962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":35920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5449},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8964","reported_seq":9020,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.730044+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:57.730044+0000","last_peered":"2024-09-18T11:57:57.730044+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:57.730044+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8964,"log_dups_size":0,"ondisk_log_size":8964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36697298,"num_objects":8961,"num_object_clones":0,"num_object_copies":17922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8962,"num_write_kb":35839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5411}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9006","reported_seq":9215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.815408+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:57.815408+0000","last_peered":"2024-09-18T11:57:57.815408+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:57.815408+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36864000,"num_objects":9000,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5486,"num_write_kb":21944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5407}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8977","reported_seq":9058,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.291658+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:57.291658+0000","last_peered":"2024-09-18T11:57:57.291658+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:57.291658+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8977,"log_dups_size":0,"ondisk_log_size":8977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36762217,"num_objects":8976,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8976,"num_write_kb":35901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5441},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9026","reported_seq":11631,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:57.252198+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:57.252198+0000","last_peered":"2024-09-18T11:57:57.252198+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:57.252198+0000","last_undegraded":"2024-09-18T11:57:57.252198+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9026,"log_dups_size":0,"ondisk_log_size":9026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36939579,"num_objects":9021,"num_object_clones":0,"num_object_copies":18042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2262,"num_objects_unfound":0,"num_objects_dirty":9021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9023,"num_write_kb":36076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6759},{"shards":"0,2","objects":2262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8891","reported_seq":8972,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.268395+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:57.268395+0000","last_peered":"2024-09-18T11:57:57.268395+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:57.268395+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8891,"log_dups_size":0,"ondisk_log_size":8891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36402386,"num_objects":8889,"num_object_clones":0,"num_object_copies":17778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8889,"num_write_kb":35550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5296},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9044","reported_seq":9128,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.275990+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:57.275990+0000","last_peered":"2024-09-18T11:57:57.275990+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:57.275990+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9044,"log_dups_size":0,"ondisk_log_size":9044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36997540,"num_objects":9036,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9041,"num_write_kb":36135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5513},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":588476468,"num_objects":143688,"num_object_clones":0,"num_object_copies":287376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2262,"num_objects_unfound":0,"num_objects_dirty":143688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127743,"num_write_kb":510852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4763,"num_bytes_recovered":19419753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1196609536,"data_stored":1196529519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143750,"ondisk_log_size":143750,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051633,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707612,"kb_used_data":401428,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93664228,"statfs":{"total":96636764160,"available":95912169472,"internally_reserved":0,"allocated":411062272,"data_stored":410838697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.439}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":6.641},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":6.7489999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855396,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585560,"kb_used_data":279376,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93786280,"statfs":{"total":96636764160,"available":96037150720,"internally_reserved":0,"allocated":286081024,"data_stored":285784247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.246},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":3.8940000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":3.569},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":2.6960000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247860,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792980,"kb_used_data":486860,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93578860,"statfs":{"total":96636764160,"available":95824752640,"internally_reserved":0,"allocated":498544640,"data_stored":498317443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.248}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":5.4829999999999997},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":499318784,"data_stored":499315305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":286527488,"data_stored":286454429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":410763264,"data_stored":410759785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:01.829 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:02.159 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 216 20 145685 145665 2.63345 1.61328 0.0361349 0.0296572 2024-09-18T11:58:02.629 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:02.639 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:02.649 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-09-18T11:58:01.096139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":588476487,"num_objects":143690,"num_object_clones":0,"num_object_copies":287380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2262,"num_objects_unfound":0,"num_objects_dirty":143690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127745,"num_write_kb":510854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4764,"num_bytes_recovered":19419772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143753,"ondisk_log_size":143753,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2086152,"kb_used_data":1167664,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281029368,"statfs":{"total":289910292480,"available":287774072832,"internally_reserved":0,"allocated":1195687936,"data_stored":1194940387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,6,5,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17498112,"num_objects":4272,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-230,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4272,"num_write_kb":17088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002920"},"pg_stats":[{"pgid":"2.f","version":"39'9053","reported_seq":9124,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.237594+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:57:57.237594+0000","last_peered":"2024-09-18T11:57:57.237594+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:57:57.237594+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37081088,"num_objects":9053,"num_object_clones":0,"num_object_copies":18106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9053,"num_write_kb":36212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5454},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'8990","reported_seq":9148,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:57.848907+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:57:57.848907+0000","last_peered":"2024-09-18T11:57:57.848907+0000","last_clean":"2024-09-18T11:57:57.848907+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:57:57.848907+0000","last_undegraded":"2024-09-18T11:57:57.848907+0000","last_fullsized":"2024-09-18T11:57:57.848907+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":8990,"log_dups_size":0,"ondisk_log_size":8990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36798464,"num_objects":8984,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6182,"num_write_kb":24728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9036","reported_seq":9221,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.757696+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:57:57.757696+0000","last_peered":"2024-09-18T11:57:57.757696+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:57:57.757696+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9036,"log_dups_size":0,"ondisk_log_size":9036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36995072,"num_objects":9032,"num_object_clones":0,"num_object_copies":18064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5894,"num_write_kb":23576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5496}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'8900","reported_seq":8929,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:57.814320+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:57:57.814320+0000","last_peered":"2024-09-18T11:57:57.814320+0000","last_clean":"2024-09-18T11:57:57.814320+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:57:57.814320+0000","last_undegraded":"2024-09-18T11:57:57.814320+0000","last_fullsized":"2024-09-18T11:57:57.814320+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8900,"log_dups_size":0,"ondisk_log_size":8900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36454400,"num_objects":8900,"num_object_clones":0,"num_object_copies":17800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8900,"num_write_kb":35600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'8976","reported_seq":9167,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:57.257823+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:57:57.257823+0000","last_peered":"2024-09-18T11:57:57.257823+0000","last_clean":"2024-09-18T11:57:57.257823+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:57:57.257823+0000","last_undegraded":"2024-09-18T11:57:57.257823+0000","last_fullsized":"2024-09-18T11:57:57.257823+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36737024,"num_objects":8969,"num_object_clones":0,"num_object_copies":17938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6141,"num_write_kb":24564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'8944","reported_seq":9025,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.213556+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:57:57.213556+0000","last_peered":"2024-09-18T11:57:57.213556+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:57:57.213556+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8944,"log_dups_size":0,"ondisk_log_size":8944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36615378,"num_objects":8941,"num_object_clones":0,"num_object_copies":17882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8942,"num_write_kb":35759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5414},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'8943","reported_seq":9030,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.113679+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:57:57.113679+0000","last_peered":"2024-09-18T11:57:57.113679+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:57:57.113679+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8943,"log_dups_size":0,"ondisk_log_size":8943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36607186,"num_objects":8939,"num_object_clones":0,"num_object_copies":17878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8941,"num_write_kb":35752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5380},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9093","reported_seq":9318,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:57.227239+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:57:57.227239+0000","last_peered":"2024-09-18T11:57:57.227239+0000","last_clean":"2024-09-18T11:57:57.227239+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:57:57.227239+0000","last_undegraded":"2024-09-18T11:57:57.227239+0000","last_fullsized":"2024-09-18T11:57:57.227239+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9093,"log_dups_size":0,"ondisk_log_size":9093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37236736,"num_objects":9091,"num_object_clones":0,"num_object_copies":18182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":21640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'8920","reported_seq":16048,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:57:57.440046+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:57:57.440046+0000","last_peered":"2024-09-18T11:57:57.440046+0000","last_clean":"2024-09-18T11:57:57.440046+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:57:57.440046+0000","last_undegraded":"2024-09-18T11:57:57.440046+0000","last_fullsized":"2024-09-18T11:57:57.440046+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":8920,"log_dups_size":0,"ondisk_log_size":8920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36508882,"num_objects":8915,"num_object_clones":0,"num_object_copies":17830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":35656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'8987","reported_seq":9064,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.276627+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:57:57.276627+0000","last_peered":"2024-09-18T11:57:57.276627+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:57:57.276627+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8987,"log_dups_size":0,"ondisk_log_size":8987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36779218,"num_objects":8981,"num_object_clones":0,"num_object_copies":17962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":35920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5449},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'8964","reported_seq":9020,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.730044+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:57:57.730044+0000","last_peered":"2024-09-18T11:57:57.730044+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:57:57.730044+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8964,"log_dups_size":0,"ondisk_log_size":8964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36697298,"num_objects":8961,"num_object_clones":0,"num_object_copies":17922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8962,"num_write_kb":35839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5411}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9006","reported_seq":9215,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.815408+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:57:57.815408+0000","last_peered":"2024-09-18T11:57:57.815408+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:57:57.815408+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36864000,"num_objects":9000,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5486,"num_write_kb":21944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5407}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'8977","reported_seq":9058,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.291658+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:57:57.291658+0000","last_peered":"2024-09-18T11:57:57.291658+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:57:57.291658+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8977,"log_dups_size":0,"ondisk_log_size":8977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36762217,"num_objects":8976,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8976,"num_write_kb":35901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5441},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9026","reported_seq":11631,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:57:57.252198+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:57:57.252198+0000","last_peered":"2024-09-18T11:57:57.252198+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:57:57.252198+0000","last_undegraded":"2024-09-18T11:57:57.252198+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9026,"log_dups_size":0,"ondisk_log_size":9026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36939579,"num_objects":9021,"num_object_clones":0,"num_object_copies":18042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2262,"num_objects_unfound":0,"num_objects_dirty":9021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9023,"num_write_kb":36076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6759},{"shards":"0,2","objects":2262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'8891","reported_seq":8972,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.268395+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:57:57.268395+0000","last_peered":"2024-09-18T11:57:57.268395+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:57:57.268395+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":8891,"log_dups_size":0,"ondisk_log_size":8891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36402386,"num_objects":8889,"num_object_clones":0,"num_object_copies":17778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8889,"num_write_kb":35550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5296},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9044","reported_seq":9128,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:57:57.275990+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:57:57.275990+0000","last_peered":"2024-09-18T11:57:57.275990+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:57:57.275990+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9044,"log_dups_size":0,"ondisk_log_size":9044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36997540,"num_objects":9036,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9041,"num_write_kb":36135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5513},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":588476468,"num_objects":143688,"num_object_clones":0,"num_object_copies":287376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2262,"num_objects_unfound":0,"num_objects_dirty":143688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127743,"num_write_kb":510852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4763,"num_bytes_recovered":19419753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1196609536,"data_stored":1196529519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143750,"ondisk_log_size":143750,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051633,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707612,"kb_used_data":401428,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93664228,"statfs":{"total":96636764160,"available":95912169472,"internally_reserved":0,"allocated":411062272,"data_stored":410838697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.439}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":6.641},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":6.7489999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855396,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585560,"kb_used_data":279376,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93786280,"statfs":{"total":96636764160,"available":96037150720,"internally_reserved":0,"allocated":286081024,"data_stored":285784247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.246},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":3.8940000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":3.569},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":2.6960000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247860,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":792980,"kb_used_data":486860,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93578860,"statfs":{"total":96636764160,"available":95824752640,"internally_reserved":0,"allocated":498544640,"data_stored":498317443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.248}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":5.4829999999999997},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":499318784,"data_stored":499315305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":286527488,"data_stored":286454429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":410763264,"data_stored":410759785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:02.651 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:58:02.651 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'9053", 'reported_seq': 9124, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:57.237594+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:57:57.237594+0000', 'last_peered': '2024-09-18T11:57:57.237594+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:57:57.237594+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9053, 'log_dups_size': 0, 'ondisk_log_size': 9053, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37081088, 'num_objects': 9053, 'num_object_clones': 0, 'num_object_copies': 18106, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9053, 'num_write_kb': 36212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5454}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:02.651 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:58:02.651 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'9036", 'reported_seq': 9221, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:57.757696+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:57:57.757696+0000', 'last_peered': '2024-09-18T11:57:57.757696+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:57:57.757696+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 9036, 'log_dups_size': 0, 'ondisk_log_size': 9036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36995072, 'num_objects': 9032, 'num_object_clones': 0, 'num_object_copies': 18064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5894, 'num_write_kb': 23576, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 5496}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:02.651 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:58:02.651 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'8944", 'reported_seq': 9025, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:57.213556+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:57:57.213556+0000', 'last_peered': '2024-09-18T11:57:57.213556+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:57:57.213556+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8944, 'log_dups_size': 0, 'ondisk_log_size': 8944, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36615378, 'num_objects': 8941, 'num_object_clones': 0, 'num_object_copies': 17882, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8942, 'num_write_kb': 35759, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5414}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:02.651 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:58:02.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'8943", 'reported_seq': 9030, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:57.113679+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:57:57.113679+0000', 'last_peered': '2024-09-18T11:57:57.113679+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:57:57.113679+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8943, 'log_dups_size': 0, 'ondisk_log_size': 8943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36607186, 'num_objects': 8939, 'num_object_clones': 0, 'num_object_copies': 17878, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8941, 'num_write_kb': 35752, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5380}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:02.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:58:02.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'8987", 'reported_seq': 9064, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:57.276627+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:57:57.276627+0000', 'last_peered': '2024-09-18T11:57:57.276627+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:57:57.276627+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8987, 'log_dups_size': 0, 'ondisk_log_size': 8987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36779218, 'num_objects': 8981, 'num_object_clones': 0, 'num_object_copies': 17962, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8985, 'num_write_kb': 35920, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5449}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:02.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:58:02.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'8964", 'reported_seq': 9020, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:57.730044+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:57:57.730044+0000', 'last_peered': '2024-09-18T11:57:57.730044+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:57:57.730044+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8964, 'log_dups_size': 0, 'ondisk_log_size': 8964, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36697298, 'num_objects': 8961, 'num_object_clones': 0, 'num_object_copies': 17922, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8961, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8962, 'num_write_kb': 35839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 5411}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:02.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:58:02.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'9006", 'reported_seq': 9215, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:57.815408+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:57:57.815408+0000', 'last_peered': '2024-09-18T11:57:57.815408+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:57:57.815408+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 9006, 'log_dups_size': 0, 'ondisk_log_size': 9006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36864000, 'num_objects': 9000, 'num_object_clones': 0, 'num_object_copies': 18000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9000, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5486, 'num_write_kb': 21944, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 5407}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:02.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:58:02.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'8977", 'reported_seq': 9058, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:57.291658+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:57:57.291658+0000', 'last_peered': '2024-09-18T11:57:57.291658+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:57:57.291658+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8977, 'log_dups_size': 0, 'ondisk_log_size': 8977, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36762217, 'num_objects': 8976, 'num_object_clones': 0, 'num_object_copies': 17952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8976, 'num_write_kb': 35901, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5441}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:02.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:58:02.653 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'9026", 'reported_seq': 11631, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:57:57.252198+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:57:57.252198+0000', 'last_peered': '2024-09-18T11:57:57.252198+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:57:57.252198+0000', 'last_undegraded': '2024-09-18T11:57:57.252198+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9026, 'log_dups_size': 0, 'ondisk_log_size': 9026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36939579, 'num_objects': 9021, 'num_object_clones': 0, 'num_object_copies': 18042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2262, 'num_objects_unfound': 0, 'num_objects_dirty': 9021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9023, 'num_write_kb': 36076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1263, 'num_bytes_recovered': 5160960, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6759}, {'shards': '0,2', 'objects': 2262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:02.653 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:58:02.653 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'8891", 'reported_seq': 8972, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:57.268395+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:57:57.268395+0000', 'last_peered': '2024-09-18T11:57:57.268395+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:57:57.268395+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 8891, 'log_dups_size': 0, 'ondisk_log_size': 8891, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36402386, 'num_objects': 8889, 'num_object_clones': 0, 'num_object_copies': 17778, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8889, 'num_write_kb': 35550, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5296}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:02.653 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:58:02.653 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'9044", 'reported_seq': 9128, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:57:57.275990+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:57:57.275990+0000', 'last_peered': '2024-09-18T11:57:57.275990+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:57:57.275990+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9044, 'log_dups_size': 0, 'ondisk_log_size': 9044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36997540, 'num_objects': 9036, 'num_object_clones': 0, 'num_object_copies': 18072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9036, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9041, 'num_write_kb': 36135, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5513}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:02.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:03.159 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 217 20 146089 146069 2.62859 1.57812 0.0229278 0.0297094 2024-09-18T11:58:03.648 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:03.648 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/291666 objects degraded (12.193%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":283,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":145833,"data_bytes":597254215,"bytes_used":2167205888,"bytes_avail":287743086592,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":291666,"degraded_ratio":0.12193399299198399,"misplaced_objects":2150,"misplaced_total":291666,"misplaced_ratio":0.0073714454204466765,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":116702,"recovering_keys_per_sec":0,"num_objects_recovered":342,"num_bytes_recovered":1400832,"num_keys_recovered":0,"write_bytes_sec":2189025,"read_op_per_sec":0,"write_op_per_sec":534},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:03.693 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:03.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:04.159 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 218 20 146488 146468 2.62368 1.55859 0.0701021 0.0297685 2024-09-18T11:58:04.355 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:04.397 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:04.442 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-18T11:58:03.097188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597254215,"num_objects":145833,"num_object_clones":0,"num_object_copies":291666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":145833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129888,"num_write_kb":519426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4876,"num_bytes_recovered":19878524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145896,"ondisk_log_size":145896,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2116412,"kb_used_data":1185444,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":280999108,"statfs":{"total":289910292480,"available":287743086592,"internally_reserved":0,"allocated":1213894656,"data_stored":1213147107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,1,9,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26275840,"num_objects":6415,"num_object_clones":0,"num_object_copies":12830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-342,"num_objects_unfound":0,"num_objects_dirty":6415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6415,"num_write_kb":25660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003439"},"pg_stats":[{"pgid":"2.f","version":"39'9198","reported_seq":9269,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.310772+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:02.310772+0000","last_peered":"2024-09-18T11:58:02.310772+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:02.310772+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9198,"log_dups_size":0,"ondisk_log_size":9198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37675008,"num_objects":9198,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9198,"num_write_kb":36792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5599},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9127","reported_seq":9285,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.762624+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:02.762624+0000","last_peered":"2024-09-18T11:58:02.762624+0000","last_clean":"2024-09-18T11:58:02.762624+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:02.762624+0000","last_undegraded":"2024-09-18T11:58:02.762624+0000","last_fullsized":"2024-09-18T11:58:02.762624+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9127,"log_dups_size":0,"ondisk_log_size":9127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37359616,"num_objects":9121,"num_object_clones":0,"num_object_copies":18242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6319,"num_write_kb":25276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9176","reported_seq":9360,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.818122+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:02.818122+0000","last_peered":"2024-09-18T11:58:02.818122+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:02.818122+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37568512,"num_objects":9172,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6034,"num_write_kb":24136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5636}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9040","reported_seq":9069,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.786518+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:02.786518+0000","last_peered":"2024-09-18T11:58:02.786518+0000","last_clean":"2024-09-18T11:58:02.786518+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:02.786518+0000","last_undegraded":"2024-09-18T11:58:02.786518+0000","last_fullsized":"2024-09-18T11:58:02.786518+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9040,"log_dups_size":0,"ondisk_log_size":9040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37027840,"num_objects":9040,"num_object_clones":0,"num_object_copies":18080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9040,"num_write_kb":36160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9110","reported_seq":9301,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.324934+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:02.324934+0000","last_peered":"2024-09-18T11:58:02.324934+0000","last_clean":"2024-09-18T11:58:02.324934+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:02.324934+0000","last_undegraded":"2024-09-18T11:58:02.324934+0000","last_fullsized":"2024-09-18T11:58:02.324934+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9110,"log_dups_size":0,"ondisk_log_size":9110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37285888,"num_objects":9103,"num_object_clones":0,"num_object_copies":18206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6275,"num_write_kb":25100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9091","reported_seq":9171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.294400+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:02.294400+0000","last_peered":"2024-09-18T11:58:02.294400+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:02.294400+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9091,"log_dups_size":0,"ondisk_log_size":9091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37217490,"num_objects":9088,"num_object_clones":0,"num_object_copies":18176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":36347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5561},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9065","reported_seq":9151,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.303760+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:02.303760+0000","last_peered":"2024-09-18T11:58:02.303760+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:02.303760+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37106898,"num_objects":9061,"num_object_clones":0,"num_object_copies":18122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9063,"num_write_kb":36240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5502},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9225","reported_seq":9450,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.273576+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:02.273576+0000","last_peered":"2024-09-18T11:58:02.273576+0000","last_clean":"2024-09-18T11:58:02.273576+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:02.273576+0000","last_undegraded":"2024-09-18T11:58:02.273576+0000","last_fullsized":"2024-09-18T11:58:02.273576+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9225,"log_dups_size":0,"ondisk_log_size":9225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37777408,"num_objects":9223,"num_object_clones":0,"num_object_copies":18446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5542,"num_write_kb":22168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9050","reported_seq":16178,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.445945+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:02.445945+0000","last_peered":"2024-09-18T11:58:02.445945+0000","last_clean":"2024-09-18T11:58:02.445945+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:02.445945+0000","last_undegraded":"2024-09-18T11:58:02.445945+0000","last_fullsized":"2024-09-18T11:58:02.445945+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9050,"log_dups_size":0,"ondisk_log_size":9050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37041362,"num_objects":9045,"num_object_clones":0,"num_object_copies":18090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9048,"num_write_kb":36176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9118","reported_seq":9195,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.242757+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:02.242757+0000","last_peered":"2024-09-18T11:58:02.242757+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:02.242757+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9118,"log_dups_size":0,"ondisk_log_size":9118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37315794,"num_objects":9112,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9116,"num_write_kb":36444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5580},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9103","reported_seq":9159,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.800106+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:02.800106+0000","last_peered":"2024-09-18T11:58:02.800106+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:02.800106+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9103,"log_dups_size":0,"ondisk_log_size":9103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37266642,"num_objects":9100,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9101,"num_write_kb":36395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5550}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9139","reported_seq":9348,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.865870+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:02.865870+0000","last_peered":"2024-09-18T11:58:02.865870+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:02.865870+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9139,"log_dups_size":0,"ondisk_log_size":9139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37408768,"num_objects":9133,"num_object_clones":0,"num_object_copies":18266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5619,"num_write_kb":22476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5540}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9124","reported_seq":9206,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.275396+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:02.275396+0000","last_peered":"2024-09-18T11:58:02.275396+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:02.275396+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37364329,"num_objects":9123,"num_object_clones":0,"num_object_copies":18246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9123,"num_write_kb":36489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5588},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9166","reported_seq":11994,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:02.324223+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:02.324223+0000","last_peered":"2024-09-18T11:58:02.324223+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:02.324223+0000","last_undegraded":"2024-09-18T11:58:02.324223+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9166,"log_dups_size":0,"ondisk_log_size":9166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37513019,"num_objects":9161,"num_object_clones":0,"num_object_copies":18322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":9161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9163,"num_write_kb":36636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":5619712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7011},{"shards":"0,2","objects":2150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9002","reported_seq":9083,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.207837+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:02.207837+0000","last_peered":"2024-09-18T11:58:02.207837+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:02.207837+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9002,"log_dups_size":0,"ondisk_log_size":9002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36857042,"num_objects":9000,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9000,"num_write_kb":35994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5407},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9159","reported_seq":9244,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.308632+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:02.308632+0000","last_peered":"2024-09-18T11:58:02.308632+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:02.308632+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9159,"log_dups_size":0,"ondisk_log_size":9159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37468580,"num_objects":9151,"num_object_clones":0,"num_object_copies":18302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":36595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5628},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":597254196,"num_objects":145831,"num_object_clones":0,"num_object_copies":291662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":145831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129886,"num_write_kb":519424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4875,"num_bytes_recovered":19878505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1214640128,"data_stored":1214560111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145893,"ondisk_log_size":145893,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051634,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716084,"kb_used_data":405740,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93655756,"statfs":{"total":96636764160,"available":95903494144,"internally_reserved":0,"allocated":415477760,"data_stored":415254185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.439}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":6.641},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":6.7489999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855397,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596604,"kb_used_data":286260,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93775236,"statfs":{"total":96636764160,"available":96025841664,"internally_reserved":0,"allocated":293130240,"data_stored":292833463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.22500000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.311},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.24099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":803724,"kb_used_data":493444,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93568116,"statfs":{"total":96636764160,"available":95813750784,"internally_reserved":0,"allocated":505286656,"data_stored":505059459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,0,6,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":3.819}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.254},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":505843712,"data_stored":505840233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293593088,"data_stored":293520029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":415203328,"data_stored":415199849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:05.159 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 219 20 146884 146864 2.61876 1.54688 0.0690494 0.029825 2024-09-18T11:58:06.159 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:2024-09-18T11:58:06.160213+0000 min lat: 0.0021114 max lat: 2.57663 avg lat: 0.0298744 2024-09-18T11:58:06.159 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T11:58:06.159 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 220 20 147311 147291 2.61444 1.66797 0.0758247 0.0298744 2024-09-18T11:58:07.159 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 221 20 147751 147731 2.61039 1.71875 0.0284557 0.0299201 2024-09-18T11:58:07.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:08.128 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:08.159 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 222 20 148167 148147 2.60595 1.625 0.0562584 0.0299712 2024-09-18T11:58:08.174 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:08.181 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-18T11:58:07.098124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597254215,"num_objects":145833,"num_object_clones":0,"num_object_copies":291666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":145833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129888,"num_write_kb":519426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4876,"num_bytes_recovered":19878524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145896,"ondisk_log_size":145896,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2116412,"kb_used_data":1185444,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":280999108,"statfs":{"total":289910292480,"available":287743086592,"internally_reserved":0,"allocated":1213894656,"data_stored":1213147107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,1,9,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17588224,"num_objects":4294,"num_object_clones":0,"num_object_copies":8588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-234,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4294,"num_write_kb":17176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003388"},"pg_stats":[{"pgid":"2.f","version":"39'9198","reported_seq":9269,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.310772+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:02.310772+0000","last_peered":"2024-09-18T11:58:02.310772+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:02.310772+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9198,"log_dups_size":0,"ondisk_log_size":9198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37675008,"num_objects":9198,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9198,"num_write_kb":36792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5599},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9127","reported_seq":9285,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.762624+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:02.762624+0000","last_peered":"2024-09-18T11:58:02.762624+0000","last_clean":"2024-09-18T11:58:02.762624+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:02.762624+0000","last_undegraded":"2024-09-18T11:58:02.762624+0000","last_fullsized":"2024-09-18T11:58:02.762624+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9127,"log_dups_size":0,"ondisk_log_size":9127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37359616,"num_objects":9121,"num_object_clones":0,"num_object_copies":18242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6319,"num_write_kb":25276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9176","reported_seq":9360,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.818122+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:02.818122+0000","last_peered":"2024-09-18T11:58:02.818122+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:02.818122+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37568512,"num_objects":9172,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6034,"num_write_kb":24136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5636}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9040","reported_seq":9069,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.786518+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:02.786518+0000","last_peered":"2024-09-18T11:58:02.786518+0000","last_clean":"2024-09-18T11:58:02.786518+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:02.786518+0000","last_undegraded":"2024-09-18T11:58:02.786518+0000","last_fullsized":"2024-09-18T11:58:02.786518+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9040,"log_dups_size":0,"ondisk_log_size":9040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37027840,"num_objects":9040,"num_object_clones":0,"num_object_copies":18080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9040,"num_write_kb":36160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9110","reported_seq":9301,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.324934+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:02.324934+0000","last_peered":"2024-09-18T11:58:02.324934+0000","last_clean":"2024-09-18T11:58:02.324934+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:02.324934+0000","last_undegraded":"2024-09-18T11:58:02.324934+0000","last_fullsized":"2024-09-18T11:58:02.324934+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9110,"log_dups_size":0,"ondisk_log_size":9110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37285888,"num_objects":9103,"num_object_clones":0,"num_object_copies":18206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6275,"num_write_kb":25100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9091","reported_seq":9171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.294400+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:02.294400+0000","last_peered":"2024-09-18T11:58:02.294400+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:02.294400+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9091,"log_dups_size":0,"ondisk_log_size":9091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37217490,"num_objects":9088,"num_object_clones":0,"num_object_copies":18176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":36347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5561},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9065","reported_seq":9151,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.303760+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:02.303760+0000","last_peered":"2024-09-18T11:58:02.303760+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:02.303760+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37106898,"num_objects":9061,"num_object_clones":0,"num_object_copies":18122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9063,"num_write_kb":36240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5502},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9225","reported_seq":9450,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.273576+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:02.273576+0000","last_peered":"2024-09-18T11:58:02.273576+0000","last_clean":"2024-09-18T11:58:02.273576+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:02.273576+0000","last_undegraded":"2024-09-18T11:58:02.273576+0000","last_fullsized":"2024-09-18T11:58:02.273576+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9225,"log_dups_size":0,"ondisk_log_size":9225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37777408,"num_objects":9223,"num_object_clones":0,"num_object_copies":18446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5542,"num_write_kb":22168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9050","reported_seq":16178,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.445945+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:02.445945+0000","last_peered":"2024-09-18T11:58:02.445945+0000","last_clean":"2024-09-18T11:58:02.445945+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:02.445945+0000","last_undegraded":"2024-09-18T11:58:02.445945+0000","last_fullsized":"2024-09-18T11:58:02.445945+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9050,"log_dups_size":0,"ondisk_log_size":9050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37041362,"num_objects":9045,"num_object_clones":0,"num_object_copies":18090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9048,"num_write_kb":36176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9118","reported_seq":9195,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.242757+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:02.242757+0000","last_peered":"2024-09-18T11:58:02.242757+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:02.242757+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9118,"log_dups_size":0,"ondisk_log_size":9118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37315794,"num_objects":9112,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9116,"num_write_kb":36444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5580},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9103","reported_seq":9159,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.800106+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:02.800106+0000","last_peered":"2024-09-18T11:58:02.800106+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:02.800106+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9103,"log_dups_size":0,"ondisk_log_size":9103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37266642,"num_objects":9100,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9101,"num_write_kb":36395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5550}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9139","reported_seq":9348,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.865870+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:02.865870+0000","last_peered":"2024-09-18T11:58:02.865870+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:02.865870+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9139,"log_dups_size":0,"ondisk_log_size":9139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37408768,"num_objects":9133,"num_object_clones":0,"num_object_copies":18266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5619,"num_write_kb":22476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5540}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9124","reported_seq":9206,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.275396+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:02.275396+0000","last_peered":"2024-09-18T11:58:02.275396+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:02.275396+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37364329,"num_objects":9123,"num_object_clones":0,"num_object_copies":18246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9123,"num_write_kb":36489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5588},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9166","reported_seq":11994,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:02.324223+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:02.324223+0000","last_peered":"2024-09-18T11:58:02.324223+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:02.324223+0000","last_undegraded":"2024-09-18T11:58:02.324223+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9166,"log_dups_size":0,"ondisk_log_size":9166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37513019,"num_objects":9161,"num_object_clones":0,"num_object_copies":18322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":9161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9163,"num_write_kb":36636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":5619712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7011},{"shards":"0,2","objects":2150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9002","reported_seq":9083,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.207837+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:02.207837+0000","last_peered":"2024-09-18T11:58:02.207837+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:02.207837+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9002,"log_dups_size":0,"ondisk_log_size":9002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36857042,"num_objects":9000,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9000,"num_write_kb":35994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5407},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9159","reported_seq":9244,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.308632+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:02.308632+0000","last_peered":"2024-09-18T11:58:02.308632+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:02.308632+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9159,"log_dups_size":0,"ondisk_log_size":9159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37468580,"num_objects":9151,"num_object_clones":0,"num_object_copies":18302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":36595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5628},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":597254196,"num_objects":145831,"num_object_clones":0,"num_object_copies":291662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":145831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129886,"num_write_kb":519424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4875,"num_bytes_recovered":19878505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1214640128,"data_stored":1214560111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145893,"ondisk_log_size":145893,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051634,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716084,"kb_used_data":405740,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93655756,"statfs":{"total":96636764160,"available":95903494144,"internally_reserved":0,"allocated":415477760,"data_stored":415254185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.439}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":6.641},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":6.7489999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855397,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596604,"kb_used_data":286260,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93775236,"statfs":{"total":96636764160,"available":96025841664,"internally_reserved":0,"allocated":293130240,"data_stored":292833463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.22500000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.311},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.24099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":803724,"kb_used_data":493444,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93568116,"statfs":{"total":96636764160,"available":95813750784,"internally_reserved":0,"allocated":505286656,"data_stored":505059459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,0,6,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":3.819}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.254},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":505843712,"data_stored":505840233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293593088,"data_stored":293520029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":415203328,"data_stored":415199849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:08.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:08.958 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:08.978 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-18T11:58:07.098124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597254215,"num_objects":145833,"num_object_clones":0,"num_object_copies":291666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":145833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129888,"num_write_kb":519426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4876,"num_bytes_recovered":19878524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145896,"ondisk_log_size":145896,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2116412,"kb_used_data":1185444,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":280999108,"statfs":{"total":289910292480,"available":287743086592,"internally_reserved":0,"allocated":1213894656,"data_stored":1213147107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,1,9,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17588224,"num_objects":4294,"num_object_clones":0,"num_object_copies":8588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-234,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4294,"num_write_kb":17176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003388"},"pg_stats":[{"pgid":"2.f","version":"39'9198","reported_seq":9269,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.310772+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:02.310772+0000","last_peered":"2024-09-18T11:58:02.310772+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:02.310772+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9198,"log_dups_size":0,"ondisk_log_size":9198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37675008,"num_objects":9198,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9198,"num_write_kb":36792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5599},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9127","reported_seq":9285,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.762624+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:02.762624+0000","last_peered":"2024-09-18T11:58:02.762624+0000","last_clean":"2024-09-18T11:58:02.762624+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:02.762624+0000","last_undegraded":"2024-09-18T11:58:02.762624+0000","last_fullsized":"2024-09-18T11:58:02.762624+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9127,"log_dups_size":0,"ondisk_log_size":9127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37359616,"num_objects":9121,"num_object_clones":0,"num_object_copies":18242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6319,"num_write_kb":25276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9176","reported_seq":9360,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.818122+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:02.818122+0000","last_peered":"2024-09-18T11:58:02.818122+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:02.818122+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9176,"log_dups_size":0,"ondisk_log_size":9176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37568512,"num_objects":9172,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6034,"num_write_kb":24136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5636}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9040","reported_seq":9069,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.786518+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:02.786518+0000","last_peered":"2024-09-18T11:58:02.786518+0000","last_clean":"2024-09-18T11:58:02.786518+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:02.786518+0000","last_undegraded":"2024-09-18T11:58:02.786518+0000","last_fullsized":"2024-09-18T11:58:02.786518+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9040,"log_dups_size":0,"ondisk_log_size":9040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37027840,"num_objects":9040,"num_object_clones":0,"num_object_copies":18080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9040,"num_write_kb":36160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9110","reported_seq":9301,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.324934+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:02.324934+0000","last_peered":"2024-09-18T11:58:02.324934+0000","last_clean":"2024-09-18T11:58:02.324934+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:02.324934+0000","last_undegraded":"2024-09-18T11:58:02.324934+0000","last_fullsized":"2024-09-18T11:58:02.324934+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9110,"log_dups_size":0,"ondisk_log_size":9110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37285888,"num_objects":9103,"num_object_clones":0,"num_object_copies":18206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6275,"num_write_kb":25100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9091","reported_seq":9171,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.294400+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:02.294400+0000","last_peered":"2024-09-18T11:58:02.294400+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:02.294400+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9091,"log_dups_size":0,"ondisk_log_size":9091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37217490,"num_objects":9088,"num_object_clones":0,"num_object_copies":18176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":36347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5561},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9065","reported_seq":9151,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.303760+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:02.303760+0000","last_peered":"2024-09-18T11:58:02.303760+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:02.303760+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37106898,"num_objects":9061,"num_object_clones":0,"num_object_copies":18122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9063,"num_write_kb":36240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5502},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9225","reported_seq":9450,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.273576+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:02.273576+0000","last_peered":"2024-09-18T11:58:02.273576+0000","last_clean":"2024-09-18T11:58:02.273576+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:02.273576+0000","last_undegraded":"2024-09-18T11:58:02.273576+0000","last_fullsized":"2024-09-18T11:58:02.273576+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9225,"log_dups_size":0,"ondisk_log_size":9225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37777408,"num_objects":9223,"num_object_clones":0,"num_object_copies":18446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5542,"num_write_kb":22168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9050","reported_seq":16178,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:02.445945+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:02.445945+0000","last_peered":"2024-09-18T11:58:02.445945+0000","last_clean":"2024-09-18T11:58:02.445945+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:02.445945+0000","last_undegraded":"2024-09-18T11:58:02.445945+0000","last_fullsized":"2024-09-18T11:58:02.445945+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9050,"log_dups_size":0,"ondisk_log_size":9050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37041362,"num_objects":9045,"num_object_clones":0,"num_object_copies":18090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9048,"num_write_kb":36176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9118","reported_seq":9195,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.242757+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:02.242757+0000","last_peered":"2024-09-18T11:58:02.242757+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:02.242757+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9118,"log_dups_size":0,"ondisk_log_size":9118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37315794,"num_objects":9112,"num_object_clones":0,"num_object_copies":18224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9116,"num_write_kb":36444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5580},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9103","reported_seq":9159,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.800106+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:02.800106+0000","last_peered":"2024-09-18T11:58:02.800106+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:02.800106+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9103,"log_dups_size":0,"ondisk_log_size":9103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37266642,"num_objects":9100,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9101,"num_write_kb":36395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5550}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9139","reported_seq":9348,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.865870+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:02.865870+0000","last_peered":"2024-09-18T11:58:02.865870+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:02.865870+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9139,"log_dups_size":0,"ondisk_log_size":9139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37408768,"num_objects":9133,"num_object_clones":0,"num_object_copies":18266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5619,"num_write_kb":22476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5540}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9124","reported_seq":9206,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.275396+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:02.275396+0000","last_peered":"2024-09-18T11:58:02.275396+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:02.275396+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9124,"log_dups_size":0,"ondisk_log_size":9124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37364329,"num_objects":9123,"num_object_clones":0,"num_object_copies":18246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9123,"num_write_kb":36489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5588},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9166","reported_seq":11994,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:02.324223+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:02.324223+0000","last_peered":"2024-09-18T11:58:02.324223+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:02.324223+0000","last_undegraded":"2024-09-18T11:58:02.324223+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9166,"log_dups_size":0,"ondisk_log_size":9166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37513019,"num_objects":9161,"num_object_clones":0,"num_object_copies":18322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":9161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9163,"num_write_kb":36636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":5619712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7011},{"shards":"0,2","objects":2150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9002","reported_seq":9083,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.207837+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:02.207837+0000","last_peered":"2024-09-18T11:58:02.207837+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:02.207837+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9002,"log_dups_size":0,"ondisk_log_size":9002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36857042,"num_objects":9000,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9000,"num_write_kb":35994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5407},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9159","reported_seq":9244,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:02.308632+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:02.308632+0000","last_peered":"2024-09-18T11:58:02.308632+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:02.308632+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9159,"log_dups_size":0,"ondisk_log_size":9159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37468580,"num_objects":9151,"num_object_clones":0,"num_object_copies":18302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":36595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5628},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":597254196,"num_objects":145831,"num_object_clones":0,"num_object_copies":291662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":145831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129886,"num_write_kb":519424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4875,"num_bytes_recovered":19878505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1214640128,"data_stored":1214560111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145893,"ondisk_log_size":145893,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051634,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716084,"kb_used_data":405740,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93655756,"statfs":{"total":96636764160,"available":95903494144,"internally_reserved":0,"allocated":415477760,"data_stored":415254185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.439}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":6.641},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":6.7489999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855397,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596604,"kb_used_data":286260,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93775236,"statfs":{"total":96636764160,"available":96025841664,"internally_reserved":0,"allocated":293130240,"data_stored":292833463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.22500000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.311},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.24099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":803724,"kb_used_data":493444,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93568116,"statfs":{"total":96636764160,"available":95813750784,"internally_reserved":0,"allocated":505286656,"data_stored":505059459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,0,6,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":3.819}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.254},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":505843712,"data_stored":505840233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":293593088,"data_stored":293520029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":415203328,"data_stored":415199849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:08.979 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:58:08.979 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'9198", 'reported_seq': 9269, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:02.310772+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:58:02.310772+0000', 'last_peered': '2024-09-18T11:58:02.310772+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:58:02.310772+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9198, 'log_dups_size': 0, 'ondisk_log_size': 9198, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37675008, 'num_objects': 9198, 'num_object_clones': 0, 'num_object_copies': 18396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9198, 'num_write_kb': 36792, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5599}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:08.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:58:08.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'9176", 'reported_seq': 9360, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:02.818122+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:58:02.818122+0000', 'last_peered': '2024-09-18T11:58:02.818122+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:58:02.818122+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 9176, 'log_dups_size': 0, 'ondisk_log_size': 9176, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37568512, 'num_objects': 9172, 'num_object_clones': 0, 'num_object_copies': 18344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6034, 'num_write_kb': 24136, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 5636}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:08.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:58:08.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'9091", 'reported_seq': 9171, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:02.294400+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:58:02.294400+0000', 'last_peered': '2024-09-18T11:58:02.294400+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:58:02.294400+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9091, 'log_dups_size': 0, 'ondisk_log_size': 9091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37217490, 'num_objects': 9088, 'num_object_clones': 0, 'num_object_copies': 18176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9089, 'num_write_kb': 36347, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5561}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:08.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:58:08.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'9065", 'reported_seq': 9151, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:02.303760+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:58:02.303760+0000', 'last_peered': '2024-09-18T11:58:02.303760+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:58:02.303760+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9065, 'log_dups_size': 0, 'ondisk_log_size': 9065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37106898, 'num_objects': 9061, 'num_object_clones': 0, 'num_object_copies': 18122, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9063, 'num_write_kb': 36240, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5502}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:08.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:58:08.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'9118", 'reported_seq': 9195, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:02.242757+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:58:02.242757+0000', 'last_peered': '2024-09-18T11:58:02.242757+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:58:02.242757+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9118, 'log_dups_size': 0, 'ondisk_log_size': 9118, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37315794, 'num_objects': 9112, 'num_object_clones': 0, 'num_object_copies': 18224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9116, 'num_write_kb': 36444, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5580}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:08.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:58:08.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'9103", 'reported_seq': 9159, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:02.800106+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:58:02.800106+0000', 'last_peered': '2024-09-18T11:58:02.800106+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:58:02.800106+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9103, 'log_dups_size': 0, 'ondisk_log_size': 9103, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37266642, 'num_objects': 9100, 'num_object_clones': 0, 'num_object_copies': 18200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9100, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9101, 'num_write_kb': 36395, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 5550}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:08.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:58:08.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'9139", 'reported_seq': 9348, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:02.865870+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:58:02.865870+0000', 'last_peered': '2024-09-18T11:58:02.865870+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:58:02.865870+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 9139, 'log_dups_size': 0, 'ondisk_log_size': 9139, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37408768, 'num_objects': 9133, 'num_object_clones': 0, 'num_object_copies': 18266, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5619, 'num_write_kb': 22476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 5540}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:08.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:58:08.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'9124", 'reported_seq': 9206, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:02.275396+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:58:02.275396+0000', 'last_peered': '2024-09-18T11:58:02.275396+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:58:02.275396+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9124, 'log_dups_size': 0, 'ondisk_log_size': 9124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37364329, 'num_objects': 9123, 'num_object_clones': 0, 'num_object_copies': 18246, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9123, 'num_write_kb': 36489, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5588}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:08.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:58:08.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'9166", 'reported_seq': 11994, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:58:02.324223+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:58:02.324223+0000', 'last_peered': '2024-09-18T11:58:02.324223+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:58:02.324223+0000', 'last_undegraded': '2024-09-18T11:58:02.324223+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9166, 'log_dups_size': 0, 'ondisk_log_size': 9166, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37513019, 'num_objects': 9161, 'num_object_clones': 0, 'num_object_copies': 18322, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2150, 'num_objects_unfound': 0, 'num_objects_dirty': 9161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9163, 'num_write_kb': 36636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1375, 'num_bytes_recovered': 5619712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7011}, {'shards': '0,2', 'objects': 2150}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:08.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:58:08.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'9002", 'reported_seq': 9083, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:02.207837+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:58:02.207837+0000', 'last_peered': '2024-09-18T11:58:02.207837+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:58:02.207837+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9002, 'log_dups_size': 0, 'ondisk_log_size': 9002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36857042, 'num_objects': 9000, 'num_object_clones': 0, 'num_object_copies': 18000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9000, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9000, 'num_write_kb': 35994, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5407}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:08.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:58:08.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'9159", 'reported_seq': 9244, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:02.308632+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:58:02.308632+0000', 'last_peered': '2024-09-18T11:58:02.308632+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:58:02.308632+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9159, 'log_dups_size': 0, 'ondisk_log_size': 9159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37468580, 'num_objects': 9151, 'num_object_clones': 0, 'num_object_copies': 18302, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9156, 'num_write_kb': 36595, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5628}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:08.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:09.159 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 223 20 148580 148560 2.6015 1.61328 0.0148575 0.0300213 2024-09-18T11:58:09.661 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:09.661 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/295798 objects degraded (12.023%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":289,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":147899,"data_bytes":605716551,"bytes_used":2150252544,"bytes_avail":287760039936,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":295798,"degraded_ratio":0.12023069797632167,"misplaced_objects":2026,"misplaced_total":295798,"misplaced_ratio":0.0068492687577333174,"recovering_objects_per_sec":29,"recovering_bytes_per_sec":122162,"recovering_keys_per_sec":0,"num_objects_recovered":358,"num_bytes_recovered":1466368,"num_keys_recovered":0,"write_bytes_sec":2170264,"read_op_per_sec":0,"write_op_per_sec":529},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:09.725 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:09.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:10.159 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 224 20 148984 148964 2.59693 1.57812 0.0148513 0.0300746 2024-09-18T11:58:10.545 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:10.556 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:10.587 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-18T11:58:09.098587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":605716551,"num_objects":147899,"num_object_clones":0,"num_object_copies":295798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2026,"num_objects_unfound":0,"num_objects_dirty":147899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131954,"num_write_kb":527690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5000,"num_bytes_recovered":20386428,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147962,"ondisk_log_size":147962,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2099856,"kb_used_data":1202168,"kb_used_omap":13,"kb_used_meta":897650,"kb_avail":281015664,"statfs":{"total":289910292480,"available":287760039936,"internally_reserved":0,"allocated":1231020032,"data_stored":1230272483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":919194130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,4,12,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26050560,"num_objects":6360,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-358,"num_objects_unfound":0,"num_objects_dirty":6360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6360,"num_write_kb":25440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":358,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003400"},"pg_stats":[{"pgid":"2.f","version":"39'9318","reported_seq":9389,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:07.258462+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:07.258462+0000","last_peered":"2024-09-18T11:58:07.258462+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:07.258462+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9318,"log_dups_size":0,"ondisk_log_size":9318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38166528,"num_objects":9318,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9318,"num_write_kb":37272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5719},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9251","reported_seq":9409,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:07.817545+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:07.817545+0000","last_peered":"2024-09-18T11:58:07.817545+0000","last_clean":"2024-09-18T11:58:07.817545+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:07.817545+0000","last_undegraded":"2024-09-18T11:58:07.817545+0000","last_fullsized":"2024-09-18T11:58:07.817545+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37867520,"num_objects":9245,"num_object_clones":0,"num_object_copies":18490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6443,"num_write_kb":25772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9299","reported_seq":9484,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:07.775706+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:07.775706+0000","last_peered":"2024-09-18T11:58:07.775706+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:07.775706+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9299,"log_dups_size":0,"ondisk_log_size":9299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38072320,"num_objects":9295,"num_object_clones":0,"num_object_copies":18590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":24628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5759}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9158","reported_seq":9187,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:07.761437+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:07.761437+0000","last_peered":"2024-09-18T11:58:07.761437+0000","last_clean":"2024-09-18T11:58:07.761437+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:07.761437+0000","last_undegraded":"2024-09-18T11:58:07.761437+0000","last_fullsized":"2024-09-18T11:58:07.761437+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9158,"log_dups_size":0,"ondisk_log_size":9158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37511168,"num_objects":9158,"num_object_clones":0,"num_object_copies":18316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9158,"num_write_kb":36632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9243","reported_seq":9434,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:07.178401+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:07.178401+0000","last_peered":"2024-09-18T11:58:07.178401+0000","last_clean":"2024-09-18T11:58:07.178401+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:07.178401+0000","last_undegraded":"2024-09-18T11:58:07.178401+0000","last_fullsized":"2024-09-18T11:58:07.178401+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9243,"log_dups_size":0,"ondisk_log_size":9243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37830656,"num_objects":9236,"num_object_clones":0,"num_object_copies":18472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6408,"num_write_kb":25632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9222","reported_seq":9303,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:07.297880+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:07.297880+0000","last_peered":"2024-09-18T11:58:07.297880+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:07.297880+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37754066,"num_objects":9219,"num_object_clones":0,"num_object_copies":18438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":36871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5692},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9184","reported_seq":9271,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:07.296559+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:07.296559+0000","last_peered":"2024-09-18T11:58:07.296559+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:07.296559+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9184,"log_dups_size":0,"ondisk_log_size":9184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37594322,"num_objects":9180,"num_object_clones":0,"num_object_copies":18360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9182,"num_write_kb":36716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5621},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9351","reported_seq":9576,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:07.311707+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:07.311707+0000","last_peered":"2024-09-18T11:58:07.311707+0000","last_clean":"2024-09-18T11:58:07.311707+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:07.311707+0000","last_undegraded":"2024-09-18T11:58:07.311707+0000","last_fullsized":"2024-09-18T11:58:07.311707+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9351,"log_dups_size":0,"ondisk_log_size":9351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38293504,"num_objects":9349,"num_object_clones":0,"num_object_copies":18698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5668,"num_write_kb":22672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9196","reported_seq":16324,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:07.415920+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:07.415920+0000","last_peered":"2024-09-18T11:58:07.415920+0000","last_clean":"2024-09-18T11:58:07.415920+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:07.415920+0000","last_undegraded":"2024-09-18T11:58:07.415920+0000","last_fullsized":"2024-09-18T11:58:07.415920+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9196,"log_dups_size":0,"ondisk_log_size":9196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37639378,"num_objects":9191,"num_object_clones":0,"num_object_copies":18382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9194,"num_write_kb":36760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9251","reported_seq":9328,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:07.298645+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:07.298645+0000","last_peered":"2024-09-18T11:58:07.298645+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:07.298645+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37860562,"num_objects":9245,"num_object_clones":0,"num_object_copies":18490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":36976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5713},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9245","reported_seq":9301,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:07.780737+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:07.780737+0000","last_peered":"2024-09-18T11:58:07.780737+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:07.780737+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9245,"log_dups_size":0,"ondisk_log_size":9245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37848274,"num_objects":9242,"num_object_clones":0,"num_object_copies":18484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9243,"num_write_kb":36963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5692}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9256","reported_seq":9465,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:07.774920+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:07.774920+0000","last_peered":"2024-09-18T11:58:07.774920+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:07.774920+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9256,"log_dups_size":0,"ondisk_log_size":9256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37888000,"num_objects":9250,"num_object_clones":0,"num_object_copies":18500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5736,"num_write_kb":22944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9258","reported_seq":9340,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:07.347840+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:07.347840+0000","last_peered":"2024-09-18T11:58:07.347840+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:07.347840+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9258,"log_dups_size":0,"ondisk_log_size":9258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37913193,"num_objects":9257,"num_object_clones":0,"num_object_copies":18514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9257,"num_write_kb":37025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5722},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9269","reported_seq":12346,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:07.334003+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:07.334003+0000","last_peered":"2024-09-18T11:58:07.334003+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:07.334003+0000","last_undegraded":"2024-09-18T11:58:07.334003+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9269,"log_dups_size":0,"ondisk_log_size":9269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37934907,"num_objects":9264,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2026,"num_objects_unfound":0,"num_objects_dirty":9264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9266,"num_write_kb":37048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":6127616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7238},{"shards":"0,2","objects":2026}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9153","reported_seq":9234,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:07.291398+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:07.291398+0000","last_peered":"2024-09-18T11:58:07.291398+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:07.291398+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37475538,"num_objects":9151,"num_object_clones":0,"num_object_copies":18302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":36598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5558},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9305","reported_seq":9388,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:07.298696+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:07.298696+0000","last_peered":"2024-09-18T11:58:07.298696+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:07.298696+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9305,"log_dups_size":0,"ondisk_log_size":9305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38066596,"num_objects":9297,"num_object_clones":0,"num_object_copies":18594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9302,"num_write_kb":37179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5774},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":605716532,"num_objects":147897,"num_object_clones":0,"num_object_copies":295794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":2026,"num_objects_unfound":0,"num_objects_dirty":147897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131952,"num_write_kb":527688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4999,"num_bytes_recovered":20386409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1232162816,"data_stored":1232082799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147959,"ondisk_log_size":147959,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051635,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707636,"kb_used_data":409772,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93664204,"statfs":{"total":96636764160,"available":95912144896,"internally_reserved":0,"allocated":419606528,"data_stored":419382953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":4.8339999999999996},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":4.0049999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.871,"15min":2.012},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":7.4080000000000004,"15min":7.4080000000000004},"last":4.9690000000000003},{"interface":"front","average":{"1min":1.196,"5min":0.93400000000000005,"15min":0.89000000000000001},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":8.0640000000000001,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":4.8780000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855398,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594960,"kb_used_data":292936,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93776880,"statfs":{"total":96636764160,"available":96027525120,"internally_reserved":0,"allocated":299966464,"data_stored":299669687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.26500000000000001,"5min":0.43099999999999999,"15min":0.45900000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51100000000000001,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.92000000000000004,"15min":0.95999999999999996},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.8660000000000001,"5min":5.7779999999999996,"15min":5.7779999999999996},"last":0.22500000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:06 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.083,"15min":0.95899999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.7629999999999999,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.311},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.65700000000000003,"15min":0.63700000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.492,"5min":6.492,"15min":6.492},"last":0.24099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247862,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797260,"kb_used_data":499460,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93574580,"statfs":{"total":96636764160,"available":95820369920,"internally_reserved":0,"allocated":511447040,"data_stored":511219843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,8,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.92300000000000004,"15min":0.99399999999999999},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":4.4109999999999996,"5min":4.6840000000000002,"15min":4.6840000000000002},"last":0.501},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.101,"15min":1.141},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3520000000000003,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.29299999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.23400000000000001},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":512184320,"data_stored":512180841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":300572672,"data_stored":300499613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":419405824,"data_stored":419402345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:11.159 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 225 20 149410 149390 2.59279 1.66406 0.0246246 0.030123 2024-09-18T11:58:12.162 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 226 20 149840 149820 2.58872 1.67969 0.0671344 0.0301719 2024-09-18T11:58:13.162 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 227 20 150280 150260 2.58489 1.71875 0.0246238 0.030216 2024-09-18T11:58:13.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:14.162 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 228 20 150687 150667 2.58052 1.58984 0.0404065 0.0302648 2024-09-18T11:58:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:14.218 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:14.244 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-18T11:58:13.103502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":614453319,"num_objects":150032,"num_object_clones":0,"num_object_copies":300064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1908,"num_objects_unfound":0,"num_objects_dirty":150032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134087,"num_write_kb":536222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5118,"num_bytes_recovered":20869756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150095,"ondisk_log_size":150095,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2133760,"kb_used_data":1219428,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":280981760,"statfs":{"total":289910292480,"available":287725322240,"internally_reserved":0,"allocated":1248694272,"data_stored":1247946723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,8,9,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25976832,"num_objects":6342,"num_object_clones":0,"num_object_copies":12684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-354,"num_objects_unfound":0,"num_objects_dirty":6342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6342,"num_write_kb":25368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":354,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007363"},"pg_stats":[{"pgid":"2.f","version":"39'9440","reported_seq":9511,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.312591+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:12.312591+0000","last_peered":"2024-09-18T11:58:12.312591+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:12.312591+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9440,"log_dups_size":0,"ondisk_log_size":9440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38666240,"num_objects":9440,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9440,"num_write_kb":37760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5841},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9392","reported_seq":9548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.892805+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:12.892805+0000","last_peered":"2024-09-18T11:58:12.892805+0000","last_clean":"2024-09-18T11:58:12.892805+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:12.892805+0000","last_undegraded":"2024-09-18T11:58:12.892805+0000","last_fullsized":"2024-09-18T11:58:12.892805+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9392,"log_dups_size":0,"ondisk_log_size":9392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38445056,"num_objects":9386,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":26336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9429","reported_seq":9613,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.860519+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:12.860519+0000","last_peered":"2024-09-18T11:58:12.860519+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:12.860519+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9429,"log_dups_size":0,"ondisk_log_size":9429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38604800,"num_objects":9425,"num_object_clones":0,"num_object_copies":18850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":25148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5889}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9286","reported_seq":9315,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.830726+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:12.830726+0000","last_peered":"2024-09-18T11:58:12.830726+0000","last_clean":"2024-09-18T11:58:12.830726+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:12.830726+0000","last_undegraded":"2024-09-18T11:58:12.830726+0000","last_fullsized":"2024-09-18T11:58:12.830726+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9286,"log_dups_size":0,"ondisk_log_size":9286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38035456,"num_objects":9286,"num_object_clones":0,"num_object_copies":18572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9286,"num_write_kb":37144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9388","reported_seq":9579,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.371710+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:12.371710+0000","last_peered":"2024-09-18T11:58:12.371710+0000","last_clean":"2024-09-18T11:58:12.371710+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:12.371710+0000","last_undegraded":"2024-09-18T11:58:12.371710+0000","last_fullsized":"2024-09-18T11:58:12.371710+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38424576,"num_objects":9381,"num_object_clones":0,"num_object_copies":18762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9349","reported_seq":9430,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.261299+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:12.261299+0000","last_peered":"2024-09-18T11:58:12.261299+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:12.261299+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9349,"log_dups_size":0,"ondisk_log_size":9349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38274258,"num_objects":9346,"num_object_clones":0,"num_object_copies":18692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9347,"num_write_kb":37379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5819},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9340","reported_seq":9427,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.350049+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:12.350049+0000","last_peered":"2024-09-18T11:58:12.350049+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:12.350049+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9340,"log_dups_size":0,"ondisk_log_size":9340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38233298,"num_objects":9336,"num_object_clones":0,"num_object_copies":18672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9338,"num_write_kb":37340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5777},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9483","reported_seq":9708,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.382378+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:12.382378+0000","last_peered":"2024-09-18T11:58:12.382378+0000","last_clean":"2024-09-18T11:58:12.382378+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:12.382378+0000","last_undegraded":"2024-09-18T11:58:12.382378+0000","last_fullsized":"2024-09-18T11:58:12.382378+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9483,"log_dups_size":0,"ondisk_log_size":9483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38834176,"num_objects":9481,"num_object_clones":0,"num_object_copies":18962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5800,"num_write_kb":23200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9344","reported_seq":16472,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.577756+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:12.577756+0000","last_peered":"2024-09-18T11:58:12.577756+0000","last_clean":"2024-09-18T11:58:12.577756+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:12.577756+0000","last_undegraded":"2024-09-18T11:58:12.577756+0000","last_fullsized":"2024-09-18T11:58:12.577756+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9344,"log_dups_size":0,"ondisk_log_size":9344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38245586,"num_objects":9339,"num_object_clones":0,"num_object_copies":18678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9342,"num_write_kb":37352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9381","reported_seq":9459,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.333730+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:12.333730+0000","last_peered":"2024-09-18T11:58:12.333730+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:12.333730+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9381,"log_dups_size":0,"ondisk_log_size":9381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38393042,"num_objects":9375,"num_object_clones":0,"num_object_copies":18750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9379,"num_write_kb":37496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5843},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9377","reported_seq":9433,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.783743+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:12.783743+0000","last_peered":"2024-09-18T11:58:12.783743+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:12.783743+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9377,"log_dups_size":0,"ondisk_log_size":9377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38388946,"num_objects":9374,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":37491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9396","reported_seq":9604,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.875761+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:12.875761+0000","last_peered":"2024-09-18T11:58:12.875761+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:12.875761+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9396,"log_dups_size":0,"ondisk_log_size":9396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38461440,"num_objects":9390,"num_object_clones":0,"num_object_copies":18780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5876,"num_write_kb":23504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5797}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9399","reported_seq":9481,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.432603+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:12.432603+0000","last_peered":"2024-09-18T11:58:12.432603+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:12.432603+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9399,"log_dups_size":0,"ondisk_log_size":9399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38490729,"num_objects":9398,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9398,"num_write_kb":37589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5863},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9406","reported_seq":12719,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:12.412629+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:12.412629+0000","last_peered":"2024-09-18T11:58:12.412629+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:12.412629+0000","last_undegraded":"2024-09-18T11:58:12.412629+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38496059,"num_objects":9401,"num_object_clones":0,"num_object_copies":18802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1908,"num_objects_unfound":0,"num_objects_dirty":9401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9403,"num_write_kb":37596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":6610944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7493},{"shards":"0,2","objects":1908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9265","reported_seq":9345,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.415577+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:12.415577+0000","last_peered":"2024-09-18T11:58:12.415577+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:12.415577+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9265,"log_dups_size":0,"ondisk_log_size":9265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37934290,"num_objects":9263,"num_object_clones":0,"num_object_copies":18526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9263,"num_write_kb":37046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5670},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9417","reported_seq":9502,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.326523+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:12.326523+0000","last_peered":"2024-09-18T11:58:12.326523+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:12.326523+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9417,"log_dups_size":0,"ondisk_log_size":9417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38525348,"num_objects":9409,"num_object_clones":0,"num_object_copies":18818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9414,"num_write_kb":37627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5886},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":614453300,"num_objects":150030,"num_object_clones":0,"num_object_copies":300060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1908,"num_objects_unfound":0,"num_objects_dirty":150030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134085,"num_write_kb":536220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5117,"num_bytes_recovered":20869737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1250050048,"data_stored":1249970031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150092,"ondisk_log_size":150092,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051636,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716148,"kb_used_data":414124,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93655692,"statfs":{"total":96636764160,"available":95903428608,"internally_reserved":0,"allocated":424062976,"data_stored":423839401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":8.4900000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.24099999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855399,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605828,"kb_used_data":299640,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93766012,"statfs":{"total":96636764160,"available":96016396288,"internally_reserved":0,"allocated":306831360,"data_stored":306534583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.223},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":6.5350000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.25800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":811784,"kb_used_data":505664,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93560056,"statfs":{"total":96636764160,"available":95805497344,"internally_reserved":0,"allocated":517799936,"data_stored":517572739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,1,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":1.482},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.252}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":1.5269999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":518639616,"data_stored":518636137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307503104,"data_stored":307430045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":423907328,"data_stored":423903849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:14.246 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:14.853 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:14.866 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:14.875 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-18T11:58:13.103502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":614453319,"num_objects":150032,"num_object_clones":0,"num_object_copies":300064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1908,"num_objects_unfound":0,"num_objects_dirty":150032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134087,"num_write_kb":536222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5118,"num_bytes_recovered":20869756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150095,"ondisk_log_size":150095,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2133760,"kb_used_data":1219428,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":280981760,"statfs":{"total":289910292480,"available":287725322240,"internally_reserved":0,"allocated":1248694272,"data_stored":1247946723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,8,9,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25976832,"num_objects":6342,"num_object_clones":0,"num_object_copies":12684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-354,"num_objects_unfound":0,"num_objects_dirty":6342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6342,"num_write_kb":25368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":354,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007363"},"pg_stats":[{"pgid":"2.f","version":"39'9440","reported_seq":9511,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.312591+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:12.312591+0000","last_peered":"2024-09-18T11:58:12.312591+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:12.312591+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9440,"log_dups_size":0,"ondisk_log_size":9440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38666240,"num_objects":9440,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9440,"num_write_kb":37760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5841},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9392","reported_seq":9548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.892805+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:12.892805+0000","last_peered":"2024-09-18T11:58:12.892805+0000","last_clean":"2024-09-18T11:58:12.892805+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:12.892805+0000","last_undegraded":"2024-09-18T11:58:12.892805+0000","last_fullsized":"2024-09-18T11:58:12.892805+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9392,"log_dups_size":0,"ondisk_log_size":9392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38445056,"num_objects":9386,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":26336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9429","reported_seq":9613,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.860519+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:12.860519+0000","last_peered":"2024-09-18T11:58:12.860519+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:12.860519+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9429,"log_dups_size":0,"ondisk_log_size":9429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38604800,"num_objects":9425,"num_object_clones":0,"num_object_copies":18850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":25148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5889}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9286","reported_seq":9315,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.830726+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:12.830726+0000","last_peered":"2024-09-18T11:58:12.830726+0000","last_clean":"2024-09-18T11:58:12.830726+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:12.830726+0000","last_undegraded":"2024-09-18T11:58:12.830726+0000","last_fullsized":"2024-09-18T11:58:12.830726+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9286,"log_dups_size":0,"ondisk_log_size":9286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38035456,"num_objects":9286,"num_object_clones":0,"num_object_copies":18572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9286,"num_write_kb":37144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9388","reported_seq":9579,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.371710+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:12.371710+0000","last_peered":"2024-09-18T11:58:12.371710+0000","last_clean":"2024-09-18T11:58:12.371710+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:12.371710+0000","last_undegraded":"2024-09-18T11:58:12.371710+0000","last_fullsized":"2024-09-18T11:58:12.371710+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38424576,"num_objects":9381,"num_object_clones":0,"num_object_copies":18762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9349","reported_seq":9430,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.261299+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:12.261299+0000","last_peered":"2024-09-18T11:58:12.261299+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:12.261299+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9349,"log_dups_size":0,"ondisk_log_size":9349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38274258,"num_objects":9346,"num_object_clones":0,"num_object_copies":18692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9347,"num_write_kb":37379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5819},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9340","reported_seq":9427,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.350049+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:12.350049+0000","last_peered":"2024-09-18T11:58:12.350049+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:12.350049+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9340,"log_dups_size":0,"ondisk_log_size":9340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38233298,"num_objects":9336,"num_object_clones":0,"num_object_copies":18672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9338,"num_write_kb":37340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5777},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9483","reported_seq":9708,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.382378+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:12.382378+0000","last_peered":"2024-09-18T11:58:12.382378+0000","last_clean":"2024-09-18T11:58:12.382378+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:12.382378+0000","last_undegraded":"2024-09-18T11:58:12.382378+0000","last_fullsized":"2024-09-18T11:58:12.382378+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9483,"log_dups_size":0,"ondisk_log_size":9483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38834176,"num_objects":9481,"num_object_clones":0,"num_object_copies":18962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5800,"num_write_kb":23200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9344","reported_seq":16472,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.577756+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:12.577756+0000","last_peered":"2024-09-18T11:58:12.577756+0000","last_clean":"2024-09-18T11:58:12.577756+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:12.577756+0000","last_undegraded":"2024-09-18T11:58:12.577756+0000","last_fullsized":"2024-09-18T11:58:12.577756+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9344,"log_dups_size":0,"ondisk_log_size":9344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38245586,"num_objects":9339,"num_object_clones":0,"num_object_copies":18678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9342,"num_write_kb":37352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9381","reported_seq":9459,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.333730+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:12.333730+0000","last_peered":"2024-09-18T11:58:12.333730+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:12.333730+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9381,"log_dups_size":0,"ondisk_log_size":9381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38393042,"num_objects":9375,"num_object_clones":0,"num_object_copies":18750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9379,"num_write_kb":37496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5843},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9377","reported_seq":9433,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.783743+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:12.783743+0000","last_peered":"2024-09-18T11:58:12.783743+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:12.783743+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9377,"log_dups_size":0,"ondisk_log_size":9377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38388946,"num_objects":9374,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":37491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9396","reported_seq":9604,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.875761+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:12.875761+0000","last_peered":"2024-09-18T11:58:12.875761+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:12.875761+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9396,"log_dups_size":0,"ondisk_log_size":9396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38461440,"num_objects":9390,"num_object_clones":0,"num_object_copies":18780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5876,"num_write_kb":23504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5797}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9399","reported_seq":9481,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.432603+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:12.432603+0000","last_peered":"2024-09-18T11:58:12.432603+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:12.432603+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9399,"log_dups_size":0,"ondisk_log_size":9399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38490729,"num_objects":9398,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9398,"num_write_kb":37589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5863},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9406","reported_seq":12719,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:12.412629+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:12.412629+0000","last_peered":"2024-09-18T11:58:12.412629+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:12.412629+0000","last_undegraded":"2024-09-18T11:58:12.412629+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38496059,"num_objects":9401,"num_object_clones":0,"num_object_copies":18802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1908,"num_objects_unfound":0,"num_objects_dirty":9401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9403,"num_write_kb":37596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":6610944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7493},{"shards":"0,2","objects":1908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9265","reported_seq":9345,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.415577+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:12.415577+0000","last_peered":"2024-09-18T11:58:12.415577+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:12.415577+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9265,"log_dups_size":0,"ondisk_log_size":9265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37934290,"num_objects":9263,"num_object_clones":0,"num_object_copies":18526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9263,"num_write_kb":37046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5670},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9417","reported_seq":9502,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.326523+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:12.326523+0000","last_peered":"2024-09-18T11:58:12.326523+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:12.326523+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9417,"log_dups_size":0,"ondisk_log_size":9417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38525348,"num_objects":9409,"num_object_clones":0,"num_object_copies":18818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9414,"num_write_kb":37627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5886},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":614453300,"num_objects":150030,"num_object_clones":0,"num_object_copies":300060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1908,"num_objects_unfound":0,"num_objects_dirty":150030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134085,"num_write_kb":536220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5117,"num_bytes_recovered":20869737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1250050048,"data_stored":1249970031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150092,"ondisk_log_size":150092,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051636,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716148,"kb_used_data":414124,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93655692,"statfs":{"total":96636764160,"available":95903428608,"internally_reserved":0,"allocated":424062976,"data_stored":423839401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":8.4900000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.24099999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855399,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605828,"kb_used_data":299640,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93766012,"statfs":{"total":96636764160,"available":96016396288,"internally_reserved":0,"allocated":306831360,"data_stored":306534583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.223},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":6.5350000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.25800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":811784,"kb_used_data":505664,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93560056,"statfs":{"total":96636764160,"available":95805497344,"internally_reserved":0,"allocated":517799936,"data_stored":517572739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,1,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":1.482},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.252}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":1.5269999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":518639616,"data_stored":518636137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307503104,"data_stored":307430045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":423907328,"data_stored":423903849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:14.876 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:58:14.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'9440", 'reported_seq': 9511, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:12.312591+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:58:12.312591+0000', 'last_peered': '2024-09-18T11:58:12.312591+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:58:12.312591+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9440, 'log_dups_size': 0, 'ondisk_log_size': 9440, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38666240, 'num_objects': 9440, 'num_object_clones': 0, 'num_object_copies': 18880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9440, 'num_write_kb': 37760, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5841}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:14.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:58:14.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'9429", 'reported_seq': 9613, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:12.860519+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:58:12.860519+0000', 'last_peered': '2024-09-18T11:58:12.860519+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:58:12.860519+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 9429, 'log_dups_size': 0, 'ondisk_log_size': 9429, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38604800, 'num_objects': 9425, 'num_object_clones': 0, 'num_object_copies': 18850, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6287, 'num_write_kb': 25148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 5889}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:14.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:58:14.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'9349", 'reported_seq': 9430, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:12.261299+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:58:12.261299+0000', 'last_peered': '2024-09-18T11:58:12.261299+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:58:12.261299+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9349, 'log_dups_size': 0, 'ondisk_log_size': 9349, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38274258, 'num_objects': 9346, 'num_object_clones': 0, 'num_object_copies': 18692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9347, 'num_write_kb': 37379, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5819}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:14.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:58:14.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'9340", 'reported_seq': 9427, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:12.350049+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:58:12.350049+0000', 'last_peered': '2024-09-18T11:58:12.350049+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:58:12.350049+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9340, 'log_dups_size': 0, 'ondisk_log_size': 9340, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38233298, 'num_objects': 9336, 'num_object_clones': 0, 'num_object_copies': 18672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9336, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9338, 'num_write_kb': 37340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5777}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:14.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:58:14.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'9381", 'reported_seq': 9459, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:12.333730+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:58:12.333730+0000', 'last_peered': '2024-09-18T11:58:12.333730+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:58:12.333730+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9381, 'log_dups_size': 0, 'ondisk_log_size': 9381, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38393042, 'num_objects': 9375, 'num_object_clones': 0, 'num_object_copies': 18750, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9375, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9379, 'num_write_kb': 37496, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5843}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:14.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:58:14.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'9377", 'reported_seq': 9433, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:12.783743+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:58:12.783743+0000', 'last_peered': '2024-09-18T11:58:12.783743+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:58:12.783743+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9377, 'log_dups_size': 0, 'ondisk_log_size': 9377, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38388946, 'num_objects': 9374, 'num_object_clones': 0, 'num_object_copies': 18748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9374, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9375, 'num_write_kb': 37491, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 5824}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:14.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:58:14.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'9396", 'reported_seq': 9604, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:12.875761+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:58:12.875761+0000', 'last_peered': '2024-09-18T11:58:12.875761+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:58:12.875761+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 9396, 'log_dups_size': 0, 'ondisk_log_size': 9396, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38461440, 'num_objects': 9390, 'num_object_clones': 0, 'num_object_copies': 18780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5876, 'num_write_kb': 23504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 5797}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:14.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:58:14.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'9399", 'reported_seq': 9481, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:12.432603+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:58:12.432603+0000', 'last_peered': '2024-09-18T11:58:12.432603+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:58:12.432603+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9399, 'log_dups_size': 0, 'ondisk_log_size': 9399, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38490729, 'num_objects': 9398, 'num_object_clones': 0, 'num_object_copies': 18796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9398, 'num_write_kb': 37589, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5863}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:14.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:58:14.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'9406", 'reported_seq': 12719, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:58:12.412629+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:58:12.412629+0000', 'last_peered': '2024-09-18T11:58:12.412629+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:58:12.412629+0000', 'last_undegraded': '2024-09-18T11:58:12.412629+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9406, 'log_dups_size': 0, 'ondisk_log_size': 9406, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38496059, 'num_objects': 9401, 'num_object_clones': 0, 'num_object_copies': 18802, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1908, 'num_objects_unfound': 0, 'num_objects_dirty': 9401, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9403, 'num_write_kb': 37596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1617, 'num_bytes_recovered': 6610944, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7493}, {'shards': '0,2', 'objects': 1908}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:14.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:58:14.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'9265", 'reported_seq': 9345, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:12.415577+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:58:12.415577+0000', 'last_peered': '2024-09-18T11:58:12.415577+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:58:12.415577+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9265, 'log_dups_size': 0, 'ondisk_log_size': 9265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37934290, 'num_objects': 9263, 'num_object_clones': 0, 'num_object_copies': 18526, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9263, 'num_write_kb': 37046, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5670}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:14.878 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:58:14.879 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'9417", 'reported_seq': 9502, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:12.326523+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:58:12.326523+0000', 'last_peered': '2024-09-18T11:58:12.326523+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:58:12.326523+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9417, 'log_dups_size': 0, 'ondisk_log_size': 9417, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38525348, 'num_objects': 9409, 'num_object_clones': 0, 'num_object_copies': 18818, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9414, 'num_write_kb': 37627, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5886}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:14.879 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:15.162 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 229 20 151082 151062 2.57599 1.54297 0.114423 0.0303192 2024-09-18T11:58:15.776 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:15.776 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/300064 objects degraded (11.852%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":295,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":150032,"data_bytes":614453319,"bytes_used":2184970240,"bytes_avail":287725322240,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":300064,"degraded_ratio":0.1185213821051509,"misplaced_objects":1908,"misplaced_total":300064,"misplaced_ratio":0.0063586434893889302,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":82534,"recovering_keys_per_sec":0,"num_objects_recovered":242,"num_bytes_recovered":991232,"num_keys_recovered":0,"write_bytes_sec":1432071,"read_op_per_sec":0,"write_op_per_sec":349},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:15.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:15.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:16.162 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 230 20 151468 151448 2.57135 1.50781 0.104344 0.0303731 2024-09-18T11:58:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:16.406 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:16.410 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-18T11:58:15.107138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":614453319,"num_objects":150032,"num_object_clones":0,"num_object_copies":300064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1908,"num_objects_unfound":0,"num_objects_dirty":150032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134087,"num_write_kb":536222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5118,"num_bytes_recovered":20869756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150095,"ondisk_log_size":150095,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2133760,"kb_used_data":1219428,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":280981760,"statfs":{"total":289910292480,"available":287725322240,"internally_reserved":0,"allocated":1248694272,"data_stored":1247946723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,8,9,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17199104,"num_objects":4199,"num_object_clones":0,"num_object_copies":8398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-242,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":16796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009950"},"pg_stats":[{"pgid":"2.f","version":"39'9440","reported_seq":9511,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.312591+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:12.312591+0000","last_peered":"2024-09-18T11:58:12.312591+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:12.312591+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9440,"log_dups_size":0,"ondisk_log_size":9440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38666240,"num_objects":9440,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9440,"num_write_kb":37760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5841},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9392","reported_seq":9548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.892805+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:12.892805+0000","last_peered":"2024-09-18T11:58:12.892805+0000","last_clean":"2024-09-18T11:58:12.892805+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:12.892805+0000","last_undegraded":"2024-09-18T11:58:12.892805+0000","last_fullsized":"2024-09-18T11:58:12.892805+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9392,"log_dups_size":0,"ondisk_log_size":9392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38445056,"num_objects":9386,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":26336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9429","reported_seq":9613,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.860519+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:12.860519+0000","last_peered":"2024-09-18T11:58:12.860519+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:12.860519+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9429,"log_dups_size":0,"ondisk_log_size":9429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38604800,"num_objects":9425,"num_object_clones":0,"num_object_copies":18850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":25148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":5889}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9286","reported_seq":9315,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.830726+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:12.830726+0000","last_peered":"2024-09-18T11:58:12.830726+0000","last_clean":"2024-09-18T11:58:12.830726+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:12.830726+0000","last_undegraded":"2024-09-18T11:58:12.830726+0000","last_fullsized":"2024-09-18T11:58:12.830726+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9286,"log_dups_size":0,"ondisk_log_size":9286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38035456,"num_objects":9286,"num_object_clones":0,"num_object_copies":18572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9286,"num_write_kb":37144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9388","reported_seq":9579,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.371710+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:12.371710+0000","last_peered":"2024-09-18T11:58:12.371710+0000","last_clean":"2024-09-18T11:58:12.371710+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:12.371710+0000","last_undegraded":"2024-09-18T11:58:12.371710+0000","last_fullsized":"2024-09-18T11:58:12.371710+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9388,"log_dups_size":0,"ondisk_log_size":9388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38424576,"num_objects":9381,"num_object_clones":0,"num_object_copies":18762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":26212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9349","reported_seq":9430,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.261299+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:12.261299+0000","last_peered":"2024-09-18T11:58:12.261299+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:12.261299+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9349,"log_dups_size":0,"ondisk_log_size":9349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38274258,"num_objects":9346,"num_object_clones":0,"num_object_copies":18692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9347,"num_write_kb":37379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5819},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9340","reported_seq":9427,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.350049+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:12.350049+0000","last_peered":"2024-09-18T11:58:12.350049+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:12.350049+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9340,"log_dups_size":0,"ondisk_log_size":9340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38233298,"num_objects":9336,"num_object_clones":0,"num_object_copies":18672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9338,"num_write_kb":37340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5777},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9483","reported_seq":9708,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.382378+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:12.382378+0000","last_peered":"2024-09-18T11:58:12.382378+0000","last_clean":"2024-09-18T11:58:12.382378+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:12.382378+0000","last_undegraded":"2024-09-18T11:58:12.382378+0000","last_fullsized":"2024-09-18T11:58:12.382378+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9483,"log_dups_size":0,"ondisk_log_size":9483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38834176,"num_objects":9481,"num_object_clones":0,"num_object_copies":18962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5800,"num_write_kb":23200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9344","reported_seq":16472,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:12.577756+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:12.577756+0000","last_peered":"2024-09-18T11:58:12.577756+0000","last_clean":"2024-09-18T11:58:12.577756+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:12.577756+0000","last_undegraded":"2024-09-18T11:58:12.577756+0000","last_fullsized":"2024-09-18T11:58:12.577756+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9344,"log_dups_size":0,"ondisk_log_size":9344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38245586,"num_objects":9339,"num_object_clones":0,"num_object_copies":18678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9342,"num_write_kb":37352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9381","reported_seq":9459,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.333730+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:12.333730+0000","last_peered":"2024-09-18T11:58:12.333730+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:12.333730+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9381,"log_dups_size":0,"ondisk_log_size":9381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38393042,"num_objects":9375,"num_object_clones":0,"num_object_copies":18750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9379,"num_write_kb":37496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5843},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9377","reported_seq":9433,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.783743+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:12.783743+0000","last_peered":"2024-09-18T11:58:12.783743+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:12.783743+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9377,"log_dups_size":0,"ondisk_log_size":9377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38388946,"num_objects":9374,"num_object_clones":0,"num_object_copies":18748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":37491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9396","reported_seq":9604,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.875761+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:12.875761+0000","last_peered":"2024-09-18T11:58:12.875761+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:12.875761+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9396,"log_dups_size":0,"ondisk_log_size":9396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38461440,"num_objects":9390,"num_object_clones":0,"num_object_copies":18780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5876,"num_write_kb":23504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5797}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9399","reported_seq":9481,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.432603+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:12.432603+0000","last_peered":"2024-09-18T11:58:12.432603+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:12.432603+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9399,"log_dups_size":0,"ondisk_log_size":9399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38490729,"num_objects":9398,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9398,"num_write_kb":37589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5863},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9406","reported_seq":12719,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:12.412629+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:12.412629+0000","last_peered":"2024-09-18T11:58:12.412629+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:12.412629+0000","last_undegraded":"2024-09-18T11:58:12.412629+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9406,"log_dups_size":0,"ondisk_log_size":9406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38496059,"num_objects":9401,"num_object_clones":0,"num_object_copies":18802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1908,"num_objects_unfound":0,"num_objects_dirty":9401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9403,"num_write_kb":37596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":6610944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7493},{"shards":"0,2","objects":1908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9265","reported_seq":9345,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.415577+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:12.415577+0000","last_peered":"2024-09-18T11:58:12.415577+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:12.415577+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9265,"log_dups_size":0,"ondisk_log_size":9265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37934290,"num_objects":9263,"num_object_clones":0,"num_object_copies":18526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9263,"num_write_kb":37046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5670},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9417","reported_seq":9502,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:12.326523+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:12.326523+0000","last_peered":"2024-09-18T11:58:12.326523+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:12.326523+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9417,"log_dups_size":0,"ondisk_log_size":9417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38525348,"num_objects":9409,"num_object_clones":0,"num_object_copies":18818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9414,"num_write_kb":37627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5886},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":614453300,"num_objects":150030,"num_object_clones":0,"num_object_copies":300060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1908,"num_objects_unfound":0,"num_objects_dirty":150030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134085,"num_write_kb":536220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5117,"num_bytes_recovered":20869737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1250050048,"data_stored":1249970031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150092,"ondisk_log_size":150092,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051636,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716148,"kb_used_data":414124,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93655692,"statfs":{"total":96636764160,"available":95903428608,"internally_reserved":0,"allocated":424062976,"data_stored":423839401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":8.4900000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.24099999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855399,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605828,"kb_used_data":299640,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93766012,"statfs":{"total":96636764160,"available":96016396288,"internally_reserved":0,"allocated":306831360,"data_stored":306534583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.223},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":6.5350000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.25800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":811784,"kb_used_data":505664,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93560056,"statfs":{"total":96636764160,"available":95805497344,"internally_reserved":0,"allocated":517799936,"data_stored":517572739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,1,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":1.482},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.252}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":1.5269999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":518639616,"data_stored":518636137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307503104,"data_stored":307430045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":423907328,"data_stored":423903849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:17.162 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 231 20 151874 151854 2.56708 1.58594 0.0229116 0.0304252 2024-09-18T11:58:18.162 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 232 20 152294 152274 2.56309 1.64062 0.0746763 0.0304723 2024-09-18T11:58:19.162 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 233 20 152735 152715 2.55948 1.72266 0.0270054 0.0305155 2024-09-18T11:58:19.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:20.162 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 234 20 153143 153123 2.55536 1.59375 0.114414 0.0305651 2024-09-18T11:58:20.164 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:20.182 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:20.195 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-18T11:58:19.108006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622727239,"num_objects":152052,"num_object_clones":0,"num_object_copies":304104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1772,"num_objects_unfound":0,"num_objects_dirty":152052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136107,"num_write_kb":544302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5254,"num_bytes_recovered":21426812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152115,"ondisk_log_size":152115,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2163216,"kb_used_data":1236408,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":280952304,"statfs":{"total":289910292480,"available":287695159296,"internally_reserved":0,"allocated":1266081792,"data_stored":1265334243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,11,6,9,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25473024,"num_objects":6219,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-378,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6219,"num_write_kb":24876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":1548288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009882"},"pg_stats":[{"pgid":"2.f","version":"39'9544","reported_seq":9615,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.317988+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:17.317988+0000","last_peered":"2024-09-18T11:58:17.317988+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:17.317988+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39092224,"num_objects":9544,"num_object_clones":0,"num_object_copies":19088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":38176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5945},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9504","reported_seq":9662,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.871625+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:17.871625+0000","last_peered":"2024-09-18T11:58:17.871625+0000","last_clean":"2024-09-18T11:58:17.871625+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:17.871625+0000","last_undegraded":"2024-09-18T11:58:17.871625+0000","last_fullsized":"2024-09-18T11:58:17.871625+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38903808,"num_objects":9498,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6696,"num_write_kb":26784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9553","reported_seq":9738,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.779972+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:17.779972+0000","last_peered":"2024-09-18T11:58:17.779972+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:17.779972+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9553,"log_dups_size":0,"ondisk_log_size":9553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39112704,"num_objects":9549,"num_object_clones":0,"num_object_copies":19098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6411,"num_write_kb":25644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":6013}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9402","reported_seq":9431,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.782673+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:17.782673+0000","last_peered":"2024-09-18T11:58:17.782673+0000","last_clean":"2024-09-18T11:58:17.782673+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:17.782673+0000","last_undegraded":"2024-09-18T11:58:17.782673+0000","last_fullsized":"2024-09-18T11:58:17.782673+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9402,"log_dups_size":0,"ondisk_log_size":9402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38510592,"num_objects":9402,"num_object_clones":0,"num_object_copies":18804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9402,"num_write_kb":37608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9513","reported_seq":9704,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.120009+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:17.120009+0000","last_peered":"2024-09-18T11:58:17.120009+0000","last_clean":"2024-09-18T11:58:17.120009+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:17.120009+0000","last_undegraded":"2024-09-18T11:58:17.120009+0000","last_fullsized":"2024-09-18T11:58:17.120009+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9513,"log_dups_size":0,"ondisk_log_size":9513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38936576,"num_objects":9506,"num_object_clones":0,"num_object_copies":19012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6678,"num_write_kb":26712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9492","reported_seq":9573,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.319753+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:17.319753+0000","last_peered":"2024-09-18T11:58:17.319753+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:17.319753+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9492,"log_dups_size":0,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38859986,"num_objects":9489,"num_object_clones":0,"num_object_copies":18978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9490,"num_write_kb":37951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5962},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9481","reported_seq":9568,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.384976+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:17.384976+0000","last_peered":"2024-09-18T11:58:17.384976+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:17.384976+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9481,"log_dups_size":0,"ondisk_log_size":9481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38810834,"num_objects":9477,"num_object_clones":0,"num_object_copies":18954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9479,"num_write_kb":37904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5918},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9588","reported_seq":9813,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.383808+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:17.383808+0000","last_peered":"2024-09-18T11:58:17.383808+0000","last_clean":"2024-09-18T11:58:17.383808+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:17.383808+0000","last_undegraded":"2024-09-18T11:58:17.383808+0000","last_fullsized":"2024-09-18T11:58:17.383808+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9588,"log_dups_size":0,"ondisk_log_size":9588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39264256,"num_objects":9586,"num_object_clones":0,"num_object_copies":19172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5905,"num_write_kb":23620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9487","reported_seq":16615,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.577138+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:17.577138+0000","last_peered":"2024-09-18T11:58:17.577138+0000","last_clean":"2024-09-18T11:58:17.577138+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:17.577138+0000","last_undegraded":"2024-09-18T11:58:17.577138+0000","last_fullsized":"2024-09-18T11:58:17.577138+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38831314,"num_objects":9482,"num_object_clones":0,"num_object_copies":18964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9485,"num_write_kb":37924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9520","reported_seq":9597,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.379435+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:17.379435+0000","last_peered":"2024-09-18T11:58:17.379435+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:17.379435+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9520,"log_dups_size":0,"ondisk_log_size":9520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38962386,"num_objects":9514,"num_object_clones":0,"num_object_copies":19028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9518,"num_write_kb":38052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5982},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9502","reported_seq":9558,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.854058+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:17.854058+0000","last_peered":"2024-09-18T11:58:17.854058+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:17.854058+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38900946,"num_objects":9499,"num_object_clones":0,"num_object_copies":18998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":37991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5949}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9532","reported_seq":9741,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.847464+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:17.847464+0000","last_peered":"2024-09-18T11:58:17.847464+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:17.847464+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9532,"log_dups_size":0,"ondisk_log_size":9532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39018496,"num_objects":9526,"num_object_clones":0,"num_object_copies":19052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":24048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5933}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9548","reported_seq":9630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.309890+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:17.309890+0000","last_peered":"2024-09-18T11:58:17.309890+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:17.309890+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9548,"log_dups_size":0,"ondisk_log_size":9548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39101033,"num_objects":9547,"num_object_clones":0,"num_object_copies":19094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":38185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6012},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9514","reported_seq":13099,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:17.429566+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:17.429566+0000","last_peered":"2024-09-18T11:58:17.429566+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:17.429566+0000","last_undegraded":"2024-09-18T11:58:17.429566+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9514,"log_dups_size":0,"ondisk_log_size":9514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38938427,"num_objects":9509,"num_object_clones":0,"num_object_copies":19018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1772,"num_objects_unfound":0,"num_objects_dirty":9509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9511,"num_write_kb":38028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1753,"num_bytes_recovered":7168000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7737},{"shards":"0,2","objects":1772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9382","reported_seq":9463,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.389045+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:17.389045+0000","last_peered":"2024-09-18T11:58:17.389045+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:17.389045+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9382,"log_dups_size":0,"ondisk_log_size":9382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38413522,"num_objects":9380,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":37514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5787},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9550","reported_seq":9634,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.359498+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:17.359498+0000","last_peered":"2024-09-18T11:58:17.359498+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:17.359498+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9550,"log_dups_size":0,"ondisk_log_size":9550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39070116,"num_objects":9542,"num_object_clones":0,"num_object_copies":19084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":38159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6019},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":622727220,"num_objects":152050,"num_object_clones":0,"num_object_copies":304100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1772,"num_objects_unfound":0,"num_objects_dirty":152050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136105,"num_write_kb":544300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5253,"num_bytes_recovered":21426793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1267302400,"data_stored":1267222383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152112,"ondisk_log_size":152112,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051637,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724316,"kb_used_data":418132,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93647524,"statfs":{"total":96636764160,"available":95895064576,"internally_reserved":0,"allocated":428167168,"data_stored":427943593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":1.024},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.28100000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.311},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.26600000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855400,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616912,"kb_used_data":306568,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93754928,"statfs":{"total":96636764160,"available":96005046272,"internally_reserved":0,"allocated":313925632,"data_stored":313628855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.23999999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.38300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":821988,"kb_used_data":511708,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93549852,"statfs":{"total":96636764160,"available":95795048448,"internally_reserved":0,"allocated":523988992,"data_stored":523761795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,3,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.248},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":524816384,"data_stored":524812905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":314560512,"data_stored":314487453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":427925504,"data_stored":427922025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:20.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:20.907 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:20.932 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:20.952 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-18T11:58:19.108006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622727239,"num_objects":152052,"num_object_clones":0,"num_object_copies":304104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1772,"num_objects_unfound":0,"num_objects_dirty":152052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136107,"num_write_kb":544302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5254,"num_bytes_recovered":21426812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152115,"ondisk_log_size":152115,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2163216,"kb_used_data":1236408,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":280952304,"statfs":{"total":289910292480,"available":287695159296,"internally_reserved":0,"allocated":1266081792,"data_stored":1265334243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,11,6,9,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25473024,"num_objects":6219,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-378,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6219,"num_write_kb":24876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":1548288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009882"},"pg_stats":[{"pgid":"2.f","version":"39'9544","reported_seq":9615,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.317988+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:17.317988+0000","last_peered":"2024-09-18T11:58:17.317988+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:17.317988+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39092224,"num_objects":9544,"num_object_clones":0,"num_object_copies":19088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":38176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5945},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9504","reported_seq":9662,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.871625+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:17.871625+0000","last_peered":"2024-09-18T11:58:17.871625+0000","last_clean":"2024-09-18T11:58:17.871625+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:17.871625+0000","last_undegraded":"2024-09-18T11:58:17.871625+0000","last_fullsized":"2024-09-18T11:58:17.871625+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38903808,"num_objects":9498,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6696,"num_write_kb":26784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9553","reported_seq":9738,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.779972+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:17.779972+0000","last_peered":"2024-09-18T11:58:17.779972+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:17.779972+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9553,"log_dups_size":0,"ondisk_log_size":9553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39112704,"num_objects":9549,"num_object_clones":0,"num_object_copies":19098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6411,"num_write_kb":25644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":6013}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9402","reported_seq":9431,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.782673+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:17.782673+0000","last_peered":"2024-09-18T11:58:17.782673+0000","last_clean":"2024-09-18T11:58:17.782673+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:17.782673+0000","last_undegraded":"2024-09-18T11:58:17.782673+0000","last_fullsized":"2024-09-18T11:58:17.782673+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9402,"log_dups_size":0,"ondisk_log_size":9402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38510592,"num_objects":9402,"num_object_clones":0,"num_object_copies":18804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9402,"num_write_kb":37608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9513","reported_seq":9704,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.120009+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:17.120009+0000","last_peered":"2024-09-18T11:58:17.120009+0000","last_clean":"2024-09-18T11:58:17.120009+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:17.120009+0000","last_undegraded":"2024-09-18T11:58:17.120009+0000","last_fullsized":"2024-09-18T11:58:17.120009+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9513,"log_dups_size":0,"ondisk_log_size":9513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38936576,"num_objects":9506,"num_object_clones":0,"num_object_copies":19012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6678,"num_write_kb":26712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9492","reported_seq":9573,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.319753+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:17.319753+0000","last_peered":"2024-09-18T11:58:17.319753+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:17.319753+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9492,"log_dups_size":0,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38859986,"num_objects":9489,"num_object_clones":0,"num_object_copies":18978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9490,"num_write_kb":37951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5962},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9481","reported_seq":9568,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.384976+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:17.384976+0000","last_peered":"2024-09-18T11:58:17.384976+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:17.384976+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9481,"log_dups_size":0,"ondisk_log_size":9481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38810834,"num_objects":9477,"num_object_clones":0,"num_object_copies":18954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9479,"num_write_kb":37904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5918},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9588","reported_seq":9813,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.383808+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:17.383808+0000","last_peered":"2024-09-18T11:58:17.383808+0000","last_clean":"2024-09-18T11:58:17.383808+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:17.383808+0000","last_undegraded":"2024-09-18T11:58:17.383808+0000","last_fullsized":"2024-09-18T11:58:17.383808+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9588,"log_dups_size":0,"ondisk_log_size":9588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39264256,"num_objects":9586,"num_object_clones":0,"num_object_copies":19172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5905,"num_write_kb":23620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9487","reported_seq":16615,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.577138+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:17.577138+0000","last_peered":"2024-09-18T11:58:17.577138+0000","last_clean":"2024-09-18T11:58:17.577138+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:17.577138+0000","last_undegraded":"2024-09-18T11:58:17.577138+0000","last_fullsized":"2024-09-18T11:58:17.577138+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38831314,"num_objects":9482,"num_object_clones":0,"num_object_copies":18964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9485,"num_write_kb":37924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9520","reported_seq":9597,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.379435+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:17.379435+0000","last_peered":"2024-09-18T11:58:17.379435+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:17.379435+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9520,"log_dups_size":0,"ondisk_log_size":9520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38962386,"num_objects":9514,"num_object_clones":0,"num_object_copies":19028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9518,"num_write_kb":38052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5982},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9502","reported_seq":9558,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.854058+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:17.854058+0000","last_peered":"2024-09-18T11:58:17.854058+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:17.854058+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38900946,"num_objects":9499,"num_object_clones":0,"num_object_copies":18998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":37991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5949}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9532","reported_seq":9741,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.847464+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:17.847464+0000","last_peered":"2024-09-18T11:58:17.847464+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:17.847464+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9532,"log_dups_size":0,"ondisk_log_size":9532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39018496,"num_objects":9526,"num_object_clones":0,"num_object_copies":19052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":24048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5933}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9548","reported_seq":9630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.309890+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:17.309890+0000","last_peered":"2024-09-18T11:58:17.309890+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:17.309890+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9548,"log_dups_size":0,"ondisk_log_size":9548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39101033,"num_objects":9547,"num_object_clones":0,"num_object_copies":19094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":38185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6012},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9514","reported_seq":13099,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:17.429566+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:17.429566+0000","last_peered":"2024-09-18T11:58:17.429566+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:17.429566+0000","last_undegraded":"2024-09-18T11:58:17.429566+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9514,"log_dups_size":0,"ondisk_log_size":9514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38938427,"num_objects":9509,"num_object_clones":0,"num_object_copies":19018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1772,"num_objects_unfound":0,"num_objects_dirty":9509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9511,"num_write_kb":38028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1753,"num_bytes_recovered":7168000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7737},{"shards":"0,2","objects":1772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9382","reported_seq":9463,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.389045+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:17.389045+0000","last_peered":"2024-09-18T11:58:17.389045+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:17.389045+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9382,"log_dups_size":0,"ondisk_log_size":9382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38413522,"num_objects":9380,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":37514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5787},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9550","reported_seq":9634,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.359498+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:17.359498+0000","last_peered":"2024-09-18T11:58:17.359498+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:17.359498+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9550,"log_dups_size":0,"ondisk_log_size":9550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39070116,"num_objects":9542,"num_object_clones":0,"num_object_copies":19084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":38159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6019},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":622727220,"num_objects":152050,"num_object_clones":0,"num_object_copies":304100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1772,"num_objects_unfound":0,"num_objects_dirty":152050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136105,"num_write_kb":544300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5253,"num_bytes_recovered":21426793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1267302400,"data_stored":1267222383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152112,"ondisk_log_size":152112,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051637,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724316,"kb_used_data":418132,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93647524,"statfs":{"total":96636764160,"available":95895064576,"internally_reserved":0,"allocated":428167168,"data_stored":427943593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":1.024},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.28100000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.311},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.26600000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855400,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616912,"kb_used_data":306568,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93754928,"statfs":{"total":96636764160,"available":96005046272,"internally_reserved":0,"allocated":313925632,"data_stored":313628855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.23999999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.38300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":821988,"kb_used_data":511708,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93549852,"statfs":{"total":96636764160,"available":95795048448,"internally_reserved":0,"allocated":523988992,"data_stored":523761795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,3,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.248},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":524816384,"data_stored":524812905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":314560512,"data_stored":314487453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":427925504,"data_stored":427922025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:20.953 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:58:20.953 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'9544", 'reported_seq': 9615, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:17.317988+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:58:17.317988+0000', 'last_peered': '2024-09-18T11:58:17.317988+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:58:17.317988+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9544, 'log_dups_size': 0, 'ondisk_log_size': 9544, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39092224, 'num_objects': 9544, 'num_object_clones': 0, 'num_object_copies': 19088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9544, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9544, 'num_write_kb': 38176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5945}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:20.954 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:58:20.954 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'9553", 'reported_seq': 9738, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:17.779972+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:58:17.779972+0000', 'last_peered': '2024-09-18T11:58:17.779972+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:58:17.779972+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 9553, 'log_dups_size': 0, 'ondisk_log_size': 9553, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39112704, 'num_objects': 9549, 'num_object_clones': 0, 'num_object_copies': 19098, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9549, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6411, 'num_write_kb': 25644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 6013}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:20.954 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:58:20.954 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'9492", 'reported_seq': 9573, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:17.319753+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:58:17.319753+0000', 'last_peered': '2024-09-18T11:58:17.319753+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:58:17.319753+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9492, 'log_dups_size': 0, 'ondisk_log_size': 9492, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38859986, 'num_objects': 9489, 'num_object_clones': 0, 'num_object_copies': 18978, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9490, 'num_write_kb': 37951, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5962}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:20.954 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:58:20.954 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'9481", 'reported_seq': 9568, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:17.384976+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:58:17.384976+0000', 'last_peered': '2024-09-18T11:58:17.384976+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:58:17.384976+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9481, 'log_dups_size': 0, 'ondisk_log_size': 9481, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38810834, 'num_objects': 9477, 'num_object_clones': 0, 'num_object_copies': 18954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9479, 'num_write_kb': 37904, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5918}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:20.954 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:58:20.954 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'9520", 'reported_seq': 9597, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:17.379435+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:58:17.379435+0000', 'last_peered': '2024-09-18T11:58:17.379435+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:58:17.379435+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9520, 'log_dups_size': 0, 'ondisk_log_size': 9520, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38962386, 'num_objects': 9514, 'num_object_clones': 0, 'num_object_copies': 19028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9518, 'num_write_kb': 38052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5982}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:20.954 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:58:20.954 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'9502", 'reported_seq': 9558, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:17.854058+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:58:17.854058+0000', 'last_peered': '2024-09-18T11:58:17.854058+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:58:17.854058+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9502, 'log_dups_size': 0, 'ondisk_log_size': 9502, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38900946, 'num_objects': 9499, 'num_object_clones': 0, 'num_object_copies': 18998, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9500, 'num_write_kb': 37991, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 5949}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:20.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:58:20.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'9532", 'reported_seq': 9741, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:17.847464+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:58:17.847464+0000', 'last_peered': '2024-09-18T11:58:17.847464+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:58:17.847464+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 9532, 'log_dups_size': 0, 'ondisk_log_size': 9532, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39018496, 'num_objects': 9526, 'num_object_clones': 0, 'num_object_copies': 19052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6012, 'num_write_kb': 24048, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 5933}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:20.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:58:20.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'9548", 'reported_seq': 9630, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:17.309890+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:58:17.309890+0000', 'last_peered': '2024-09-18T11:58:17.309890+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:58:17.309890+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9548, 'log_dups_size': 0, 'ondisk_log_size': 9548, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39101033, 'num_objects': 9547, 'num_object_clones': 0, 'num_object_copies': 19094, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9547, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9547, 'num_write_kb': 38185, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6012}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:20.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:58:20.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'9514", 'reported_seq': 13099, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:58:17.429566+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:58:17.429566+0000', 'last_peered': '2024-09-18T11:58:17.429566+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:58:17.429566+0000', 'last_undegraded': '2024-09-18T11:58:17.429566+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9514, 'log_dups_size': 0, 'ondisk_log_size': 9514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38938427, 'num_objects': 9509, 'num_object_clones': 0, 'num_object_copies': 19018, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1772, 'num_objects_unfound': 0, 'num_objects_dirty': 9509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9511, 'num_write_kb': 38028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1753, 'num_bytes_recovered': 7168000, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7737}, {'shards': '0,2', 'objects': 1772}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:20.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:58:20.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'9382", 'reported_seq': 9463, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:17.389045+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:58:17.389045+0000', 'last_peered': '2024-09-18T11:58:17.389045+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:58:17.389045+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9382, 'log_dups_size': 0, 'ondisk_log_size': 9382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38413522, 'num_objects': 9380, 'num_object_clones': 0, 'num_object_copies': 18760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9380, 'num_write_kb': 37514, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5787}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:20.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:58:20.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'9550", 'reported_seq': 9634, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:17.359498+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:58:17.359498+0000', 'last_peered': '2024-09-18T11:58:17.359498+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:58:17.359498+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9550, 'log_dups_size': 0, 'ondisk_log_size': 9550, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39070116, 'num_objects': 9542, 'num_object_clones': 0, 'num_object_copies': 19084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9547, 'num_write_kb': 38159, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6019}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:20.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:21.163 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 235 20 153537 153517 2.55103 1.53906 0.0110846 0.0306179 2024-09-18T11:58:21.601 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:21.601 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/304104 objects degraded (11.695%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":301,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":152052,"data_bytes":622727239,"bytes_used":2215133184,"bytes_avail":287695159296,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":304104,"degraded_ratio":0.11694683397784968,"misplaced_objects":1772,"misplaced_total":304104,"misplaced_ratio":0.0058269539368110915,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":86626,"recovering_keys_per_sec":0,"num_objects_recovered":254,"num_bytes_recovered":1040384,"num_keys_recovered":0,"write_bytes_sec":1416377,"read_op_per_sec":0,"write_op_per_sec":345},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:21.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:21.643 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:22.162 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 236 20 153917 153897 2.54651 1.48438 0.0182968 0.0306706 2024-09-18T11:58:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:22.334 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:22.348 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-18T11:58:21.108582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622727239,"num_objects":152052,"num_object_clones":0,"num_object_copies":304104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1772,"num_objects_unfound":0,"num_objects_dirty":152052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136107,"num_write_kb":544302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5254,"num_bytes_recovered":21426812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152115,"ondisk_log_size":152115,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2163216,"kb_used_data":1236408,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":280952304,"statfs":{"total":289910292480,"available":287695159296,"internally_reserved":0,"allocated":1266081792,"data_stored":1265334243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,11,6,9,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17010688,"num_objects":4153,"num_object_clones":0,"num_object_copies":8306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-254,"num_objects_unfound":0,"num_objects_dirty":4153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4153,"num_write_kb":16612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009994"},"pg_stats":[{"pgid":"2.f","version":"39'9544","reported_seq":9615,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.317988+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:17.317988+0000","last_peered":"2024-09-18T11:58:17.317988+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:17.317988+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39092224,"num_objects":9544,"num_object_clones":0,"num_object_copies":19088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":38176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5945},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9504","reported_seq":9662,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.871625+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:17.871625+0000","last_peered":"2024-09-18T11:58:17.871625+0000","last_clean":"2024-09-18T11:58:17.871625+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:17.871625+0000","last_undegraded":"2024-09-18T11:58:17.871625+0000","last_fullsized":"2024-09-18T11:58:17.871625+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38903808,"num_objects":9498,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6696,"num_write_kb":26784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9553","reported_seq":9738,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.779972+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:17.779972+0000","last_peered":"2024-09-18T11:58:17.779972+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:17.779972+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9553,"log_dups_size":0,"ondisk_log_size":9553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39112704,"num_objects":9549,"num_object_clones":0,"num_object_copies":19098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6411,"num_write_kb":25644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":6013}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9402","reported_seq":9431,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.782673+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:17.782673+0000","last_peered":"2024-09-18T11:58:17.782673+0000","last_clean":"2024-09-18T11:58:17.782673+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:17.782673+0000","last_undegraded":"2024-09-18T11:58:17.782673+0000","last_fullsized":"2024-09-18T11:58:17.782673+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9402,"log_dups_size":0,"ondisk_log_size":9402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38510592,"num_objects":9402,"num_object_clones":0,"num_object_copies":18804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9402,"num_write_kb":37608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9513","reported_seq":9704,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.120009+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:17.120009+0000","last_peered":"2024-09-18T11:58:17.120009+0000","last_clean":"2024-09-18T11:58:17.120009+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:17.120009+0000","last_undegraded":"2024-09-18T11:58:17.120009+0000","last_fullsized":"2024-09-18T11:58:17.120009+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9513,"log_dups_size":0,"ondisk_log_size":9513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38936576,"num_objects":9506,"num_object_clones":0,"num_object_copies":19012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6678,"num_write_kb":26712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9492","reported_seq":9573,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.319753+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:17.319753+0000","last_peered":"2024-09-18T11:58:17.319753+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:17.319753+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9492,"log_dups_size":0,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38859986,"num_objects":9489,"num_object_clones":0,"num_object_copies":18978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9490,"num_write_kb":37951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5962},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9481","reported_seq":9568,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.384976+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:17.384976+0000","last_peered":"2024-09-18T11:58:17.384976+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:17.384976+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9481,"log_dups_size":0,"ondisk_log_size":9481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38810834,"num_objects":9477,"num_object_clones":0,"num_object_copies":18954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9479,"num_write_kb":37904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5918},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9588","reported_seq":9813,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.383808+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:17.383808+0000","last_peered":"2024-09-18T11:58:17.383808+0000","last_clean":"2024-09-18T11:58:17.383808+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:17.383808+0000","last_undegraded":"2024-09-18T11:58:17.383808+0000","last_fullsized":"2024-09-18T11:58:17.383808+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9588,"log_dups_size":0,"ondisk_log_size":9588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39264256,"num_objects":9586,"num_object_clones":0,"num_object_copies":19172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5905,"num_write_kb":23620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9487","reported_seq":16615,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:17.577138+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:17.577138+0000","last_peered":"2024-09-18T11:58:17.577138+0000","last_clean":"2024-09-18T11:58:17.577138+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:17.577138+0000","last_undegraded":"2024-09-18T11:58:17.577138+0000","last_fullsized":"2024-09-18T11:58:17.577138+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38831314,"num_objects":9482,"num_object_clones":0,"num_object_copies":18964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9485,"num_write_kb":37924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9520","reported_seq":9597,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.379435+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:17.379435+0000","last_peered":"2024-09-18T11:58:17.379435+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:17.379435+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9520,"log_dups_size":0,"ondisk_log_size":9520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38962386,"num_objects":9514,"num_object_clones":0,"num_object_copies":19028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9518,"num_write_kb":38052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5982},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9502","reported_seq":9558,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.854058+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:17.854058+0000","last_peered":"2024-09-18T11:58:17.854058+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:17.854058+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38900946,"num_objects":9499,"num_object_clones":0,"num_object_copies":18998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":37991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":5949}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9532","reported_seq":9741,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.847464+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:17.847464+0000","last_peered":"2024-09-18T11:58:17.847464+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:17.847464+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9532,"log_dups_size":0,"ondisk_log_size":9532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39018496,"num_objects":9526,"num_object_clones":0,"num_object_copies":19052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":24048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":5933}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9548","reported_seq":9630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.309890+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:17.309890+0000","last_peered":"2024-09-18T11:58:17.309890+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:17.309890+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9548,"log_dups_size":0,"ondisk_log_size":9548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39101033,"num_objects":9547,"num_object_clones":0,"num_object_copies":19094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":38185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6012},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9514","reported_seq":13099,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:17.429566+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:17.429566+0000","last_peered":"2024-09-18T11:58:17.429566+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:17.429566+0000","last_undegraded":"2024-09-18T11:58:17.429566+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9514,"log_dups_size":0,"ondisk_log_size":9514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38938427,"num_objects":9509,"num_object_clones":0,"num_object_copies":19018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1772,"num_objects_unfound":0,"num_objects_dirty":9509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9511,"num_write_kb":38028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1753,"num_bytes_recovered":7168000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7737},{"shards":"0,2","objects":1772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9382","reported_seq":9463,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.389045+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:17.389045+0000","last_peered":"2024-09-18T11:58:17.389045+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:17.389045+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9382,"log_dups_size":0,"ondisk_log_size":9382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38413522,"num_objects":9380,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":37514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5787},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9550","reported_seq":9634,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:17.359498+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:17.359498+0000","last_peered":"2024-09-18T11:58:17.359498+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:17.359498+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9550,"log_dups_size":0,"ondisk_log_size":9550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39070116,"num_objects":9542,"num_object_clones":0,"num_object_copies":19084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":38159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6019},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":622727220,"num_objects":152050,"num_object_clones":0,"num_object_copies":304100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1772,"num_objects_unfound":0,"num_objects_dirty":152050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136105,"num_write_kb":544300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5253,"num_bytes_recovered":21426793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1267302400,"data_stored":1267222383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152112,"ondisk_log_size":152112,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051637,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724316,"kb_used_data":418132,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93647524,"statfs":{"total":96636764160,"available":95895064576,"internally_reserved":0,"allocated":428167168,"data_stored":427943593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.51000000000000001,"15min":0.42199999999999999},"min":{"1min":0.153,"5min":0.153,"15min":0.153},"max":{"1min":4.069,"5min":4.069,"15min":4.069},"last":1.024},{"interface":"front","average":{"1min":1.103,"5min":0.75600000000000001,"15min":0.44600000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.1829999999999998,"5min":9.2279999999999998,"15min":9.2279999999999998},"last":0.28100000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.311},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.26600000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855400,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616912,"kb_used_data":306568,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93754928,"statfs":{"total":96636764160,"available":96005046272,"internally_reserved":0,"allocated":313925632,"data_stored":313628855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.23999999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.38300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":821988,"kb_used_data":511708,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93549852,"statfs":{"total":96636764160,"available":95795048448,"internally_reserved":0,"allocated":523988992,"data_stored":523761795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,3,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.248},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":524816384,"data_stored":524812905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":314560512,"data_stored":314487453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":427925504,"data_stored":427922025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:23.163 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 237 20 154331 154311 2.54259 1.61719 0.0434097 0.0307177 2024-09-18T11:58:24.163 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 238 20 154763 154743 2.539 1.6875 0.0146908 0.0307621 2024-09-18T11:58:25.166 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout: 239 20 155192 155172 2.53535 1.67578 0.0699661 0.0308055 2024-09-18T11:58:25.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:26.036 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:26.048 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:26.054 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-18T11:58:25.109521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631234631,"num_objects":154129,"num_object_clones":0,"num_object_copies":308258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1657,"num_objects_unfound":0,"num_objects_dirty":154129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138184,"num_write_kb":552610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5369,"num_bytes_recovered":21897852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154192,"ondisk_log_size":154192,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2163376,"kb_used_data":1253208,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":280952144,"statfs":{"total":289910292480,"available":287694995456,"internally_reserved":0,"allocated":1283284992,"data_stored":1282537443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,7,1,8,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16781312,"num_objects":4097,"num_object_clones":0,"num_object_copies":8194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-251,"num_objects_unfound":0,"num_objects_dirty":4097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4097,"num_write_kb":16388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":1028096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006019"},"pg_stats":[{"pgid":"2.f","version":"39'9678","reported_seq":9745,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.451684+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:22.451684+0000","last_peered":"2024-09-18T11:58:22.451684+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:22.451684+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9678,"log_dups_size":0,"ondisk_log_size":9678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39641088,"num_objects":9678,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":38712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6079},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9644","reported_seq":9802,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.913197+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:22.913197+0000","last_peered":"2024-09-18T11:58:22.913197+0000","last_clean":"2024-09-18T11:58:22.913197+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:22.913197+0000","last_undegraded":"2024-09-18T11:58:22.913197+0000","last_fullsized":"2024-09-18T11:58:22.913197+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39477248,"num_objects":9638,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6836,"num_write_kb":27344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9694","reported_seq":9879,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.921115+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:22.921115+0000","last_peered":"2024-09-18T11:58:22.921115+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:22.921115+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9694,"log_dups_size":0,"ondisk_log_size":9694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39690240,"num_objects":9690,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6552,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":6154}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9538","reported_seq":9567,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.870308+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:22.870308+0000","last_peered":"2024-09-18T11:58:22.870308+0000","last_clean":"2024-09-18T11:58:22.870308+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:22.870308+0000","last_undegraded":"2024-09-18T11:58:22.870308+0000","last_fullsized":"2024-09-18T11:58:22.870308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9538,"log_dups_size":0,"ondisk_log_size":9538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39067648,"num_objects":9538,"num_object_clones":0,"num_object_copies":19076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9538,"num_write_kb":38152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9633","reported_seq":9824,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.461022+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:22.461022+0000","last_peered":"2024-09-18T11:58:22.461022+0000","last_clean":"2024-09-18T11:58:22.461022+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:22.461022+0000","last_undegraded":"2024-09-18T11:58:22.461022+0000","last_fullsized":"2024-09-18T11:58:22.461022+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9633,"log_dups_size":0,"ondisk_log_size":9633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39428096,"num_objects":9626,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6798,"num_write_kb":27192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9618","reported_seq":9699,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.442472+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:22.442472+0000","last_peered":"2024-09-18T11:58:22.442472+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:22.442472+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9618,"log_dups_size":0,"ondisk_log_size":9618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39376082,"num_objects":9615,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9616,"num_write_kb":38455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6088},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9610","reported_seq":9697,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.339135+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:22.339135+0000","last_peered":"2024-09-18T11:58:22.339135+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:22.339135+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9610,"log_dups_size":0,"ondisk_log_size":9610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39339218,"num_objects":9606,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9608,"num_write_kb":38420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6047},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9711","reported_seq":9935,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.470152+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:22.470152+0000","last_peered":"2024-09-18T11:58:22.470152+0000","last_clean":"2024-09-18T11:58:22.470152+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:22.470152+0000","last_undegraded":"2024-09-18T11:58:22.470152+0000","last_fullsized":"2024-09-18T11:58:22.470152+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9711,"log_dups_size":0,"ondisk_log_size":9711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39768064,"num_objects":9709,"num_object_clones":0,"num_object_copies":19418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6028,"num_write_kb":24112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9603","reported_seq":16731,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.703150+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:22.703150+0000","last_peered":"2024-09-18T11:58:22.703150+0000","last_clean":"2024-09-18T11:58:22.703150+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:22.703150+0000","last_undegraded":"2024-09-18T11:58:22.703150+0000","last_fullsized":"2024-09-18T11:58:22.703150+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39306450,"num_objects":9598,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":38388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9654","reported_seq":9732,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.429806+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:22.429806+0000","last_peered":"2024-09-18T11:58:22.429806+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:22.429806+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9654,"log_dups_size":0,"ondisk_log_size":9654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39511250,"num_objects":9648,"num_object_clones":0,"num_object_copies":19296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9652,"num_write_kb":38588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6116},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9638","reported_seq":9694,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.877509+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:22.877509+0000","last_peered":"2024-09-18T11:58:22.877509+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:22.877509+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9638,"log_dups_size":0,"ondisk_log_size":9638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458002,"num_objects":9635,"num_object_clones":0,"num_object_copies":19270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9636,"num_write_kb":38535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":6085}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9670","reported_seq":9879,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.850435+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:22.850435+0000","last_peered":"2024-09-18T11:58:22.850435+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:22.850435+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9670,"log_dups_size":0,"ondisk_log_size":9670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39583744,"num_objects":9664,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6150,"num_write_kb":24600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":6071}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9665","reported_seq":9745,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.333444+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:22.333444+0000","last_peered":"2024-09-18T11:58:22.333444+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:22.333444+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9665,"log_dups_size":0,"ondisk_log_size":9665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39580265,"num_objects":9664,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9664,"num_write_kb":38653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6129},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9652","reported_seq":13467,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:22.478189+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:22.478189+0000","last_peered":"2024-09-18T11:58:22.478189+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:22.478189+0000","last_undegraded":"2024-09-18T11:58:22.478189+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39503675,"num_objects":9647,"num_object_clones":0,"num_object_copies":19294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1657,"num_objects_unfound":0,"num_objects_dirty":9647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9649,"num_write_kb":38580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1868,"num_bytes_recovered":7639040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7990},{"shards":"0,2","objects":1657}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9517","reported_seq":9598,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.451753+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:22.451753+0000","last_peered":"2024-09-18T11:58:22.451753+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:22.451753+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38966482,"num_objects":9515,"num_object_clones":0,"num_object_copies":19030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9515,"num_write_kb":38054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5922},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9664","reported_seq":9749,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.331368+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:22.331368+0000","last_peered":"2024-09-18T11:58:22.331368+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:22.331368+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9664,"log_dups_size":0,"ondisk_log_size":9664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39537060,"num_objects":9656,"num_object_clones":0,"num_object_copies":19312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9661,"num_write_kb":38615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6133},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":631234612,"num_objects":154127,"num_object_clones":0,"num_object_copies":308254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1657,"num_objects_unfound":0,"num_objects_dirty":154127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138182,"num_write_kb":552608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5368,"num_bytes_recovered":21897833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1284603904,"data_stored":1284523887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154189,"ondisk_log_size":154189,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051638,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732536,"kb_used_data":422192,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93639304,"statfs":{"total":96636764160,"available":95886647296,"internally_reserved":0,"allocated":432324608,"data_stored":432101033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.24099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.214},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.26800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855401,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615140,"kb_used_data":313116,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93756700,"statfs":{"total":96636764160,"available":96006860800,"internally_reserved":0,"allocated":320630784,"data_stored":320334007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,5,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":5.3730000000000002},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.22900000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":5.577},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":5.4939999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":815700,"kb_used_data":517900,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93556140,"statfs":{"total":96636764160,"available":95801487360,"internally_reserved":0,"allocated":530329600,"data_stored":530102403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.312},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.43099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.248},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":531116032,"data_stored":531112553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":321310720,"data_stored":321237661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":432177152,"data_stored":432173673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:26.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:26.167 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Total time run: 240.018 2024-09-18T11:58:26.167 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Total writes made: 155559 2024-09-18T11:58:26.167 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Write size: 4096 2024-09-18T11:58:26.167 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Object size: 4096 2024-09-18T11:58:26.167 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Bandwidth (MB/sec): 2.53169 2024-09-18T11:58:26.168 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Stddev Bandwidth: 1.82018 2024-09-18T11:58:26.168 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Max bandwidth (MB/sec): 11.2656 2024-09-18T11:58:26.168 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Min bandwidth (MB/sec): 0 2024-09-18T11:58:26.168 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Average IOPS: 648 2024-09-18T11:58:26.168 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Stddev IOPS: 465.966 2024-09-18T11:58:26.168 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Max IOPS: 2884 2024-09-18T11:58:26.169 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Min IOPS: 0 2024-09-18T11:58:26.169 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Average Latency(s): 0.0308538 2024-09-18T11:58:26.169 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Stddev Latency(s): 0.0456528 2024-09-18T11:58:26.169 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Max latency(s): 2.57663 2024-09-18T11:58:26.169 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Min latency(s): 0.0021114 2024-09-18T11:58:26.179 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Cleaning up (deleting benchmark objects) 2024-09-18T11:58:26.749 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:26.772 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:26.788 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-18T11:58:25.109521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631234631,"num_objects":154129,"num_object_clones":0,"num_object_copies":308258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1657,"num_objects_unfound":0,"num_objects_dirty":154129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138184,"num_write_kb":552610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5369,"num_bytes_recovered":21897852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154192,"ondisk_log_size":154192,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2163376,"kb_used_data":1253208,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":280952144,"statfs":{"total":289910292480,"available":287694995456,"internally_reserved":0,"allocated":1283284992,"data_stored":1282537443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,7,1,8,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16781312,"num_objects":4097,"num_object_clones":0,"num_object_copies":8194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-251,"num_objects_unfound":0,"num_objects_dirty":4097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4097,"num_write_kb":16388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":1028096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006019"},"pg_stats":[{"pgid":"2.f","version":"39'9678","reported_seq":9745,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.451684+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:22.451684+0000","last_peered":"2024-09-18T11:58:22.451684+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:22.451684+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9678,"log_dups_size":0,"ondisk_log_size":9678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39641088,"num_objects":9678,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":38712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6079},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9644","reported_seq":9802,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.913197+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:22.913197+0000","last_peered":"2024-09-18T11:58:22.913197+0000","last_clean":"2024-09-18T11:58:22.913197+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:22.913197+0000","last_undegraded":"2024-09-18T11:58:22.913197+0000","last_fullsized":"2024-09-18T11:58:22.913197+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39477248,"num_objects":9638,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6836,"num_write_kb":27344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9694","reported_seq":9879,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.921115+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:22.921115+0000","last_peered":"2024-09-18T11:58:22.921115+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:22.921115+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9694,"log_dups_size":0,"ondisk_log_size":9694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39690240,"num_objects":9690,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6552,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":6154}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9538","reported_seq":9567,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.870308+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:22.870308+0000","last_peered":"2024-09-18T11:58:22.870308+0000","last_clean":"2024-09-18T11:58:22.870308+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:22.870308+0000","last_undegraded":"2024-09-18T11:58:22.870308+0000","last_fullsized":"2024-09-18T11:58:22.870308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9538,"log_dups_size":0,"ondisk_log_size":9538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39067648,"num_objects":9538,"num_object_clones":0,"num_object_copies":19076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9538,"num_write_kb":38152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9633","reported_seq":9824,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.461022+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:22.461022+0000","last_peered":"2024-09-18T11:58:22.461022+0000","last_clean":"2024-09-18T11:58:22.461022+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:22.461022+0000","last_undegraded":"2024-09-18T11:58:22.461022+0000","last_fullsized":"2024-09-18T11:58:22.461022+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9633,"log_dups_size":0,"ondisk_log_size":9633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39428096,"num_objects":9626,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6798,"num_write_kb":27192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9618","reported_seq":9699,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.442472+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:22.442472+0000","last_peered":"2024-09-18T11:58:22.442472+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:22.442472+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9618,"log_dups_size":0,"ondisk_log_size":9618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39376082,"num_objects":9615,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9616,"num_write_kb":38455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6088},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9610","reported_seq":9697,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.339135+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:22.339135+0000","last_peered":"2024-09-18T11:58:22.339135+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:22.339135+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9610,"log_dups_size":0,"ondisk_log_size":9610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39339218,"num_objects":9606,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9608,"num_write_kb":38420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6047},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9711","reported_seq":9935,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.470152+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:22.470152+0000","last_peered":"2024-09-18T11:58:22.470152+0000","last_clean":"2024-09-18T11:58:22.470152+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:22.470152+0000","last_undegraded":"2024-09-18T11:58:22.470152+0000","last_fullsized":"2024-09-18T11:58:22.470152+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9711,"log_dups_size":0,"ondisk_log_size":9711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39768064,"num_objects":9709,"num_object_clones":0,"num_object_copies":19418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6028,"num_write_kb":24112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9603","reported_seq":16731,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.703150+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:22.703150+0000","last_peered":"2024-09-18T11:58:22.703150+0000","last_clean":"2024-09-18T11:58:22.703150+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:22.703150+0000","last_undegraded":"2024-09-18T11:58:22.703150+0000","last_fullsized":"2024-09-18T11:58:22.703150+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39306450,"num_objects":9598,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":38388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9654","reported_seq":9732,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.429806+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:22.429806+0000","last_peered":"2024-09-18T11:58:22.429806+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:22.429806+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9654,"log_dups_size":0,"ondisk_log_size":9654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39511250,"num_objects":9648,"num_object_clones":0,"num_object_copies":19296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9652,"num_write_kb":38588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6116},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9638","reported_seq":9694,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.877509+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:22.877509+0000","last_peered":"2024-09-18T11:58:22.877509+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:22.877509+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9638,"log_dups_size":0,"ondisk_log_size":9638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458002,"num_objects":9635,"num_object_clones":0,"num_object_copies":19270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9636,"num_write_kb":38535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":6085}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9670","reported_seq":9879,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.850435+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:22.850435+0000","last_peered":"2024-09-18T11:58:22.850435+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:22.850435+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9670,"log_dups_size":0,"ondisk_log_size":9670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39583744,"num_objects":9664,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6150,"num_write_kb":24600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":6071}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9665","reported_seq":9745,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.333444+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:22.333444+0000","last_peered":"2024-09-18T11:58:22.333444+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:22.333444+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9665,"log_dups_size":0,"ondisk_log_size":9665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39580265,"num_objects":9664,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9664,"num_write_kb":38653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6129},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9652","reported_seq":13467,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:22.478189+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:22.478189+0000","last_peered":"2024-09-18T11:58:22.478189+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:22.478189+0000","last_undegraded":"2024-09-18T11:58:22.478189+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39503675,"num_objects":9647,"num_object_clones":0,"num_object_copies":19294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1657,"num_objects_unfound":0,"num_objects_dirty":9647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9649,"num_write_kb":38580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1868,"num_bytes_recovered":7639040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7990},{"shards":"0,2","objects":1657}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9517","reported_seq":9598,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.451753+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:22.451753+0000","last_peered":"2024-09-18T11:58:22.451753+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:22.451753+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38966482,"num_objects":9515,"num_object_clones":0,"num_object_copies":19030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9515,"num_write_kb":38054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5922},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9664","reported_seq":9749,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.331368+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:22.331368+0000","last_peered":"2024-09-18T11:58:22.331368+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:22.331368+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9664,"log_dups_size":0,"ondisk_log_size":9664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39537060,"num_objects":9656,"num_object_clones":0,"num_object_copies":19312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9661,"num_write_kb":38615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6133},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":631234612,"num_objects":154127,"num_object_clones":0,"num_object_copies":308254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1657,"num_objects_unfound":0,"num_objects_dirty":154127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138182,"num_write_kb":552608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5368,"num_bytes_recovered":21897833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1284603904,"data_stored":1284523887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154189,"ondisk_log_size":154189,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051638,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732536,"kb_used_data":422192,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93639304,"statfs":{"total":96636764160,"available":95886647296,"internally_reserved":0,"allocated":432324608,"data_stored":432101033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.24099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.214},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.26800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855401,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615140,"kb_used_data":313116,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93756700,"statfs":{"total":96636764160,"available":96006860800,"internally_reserved":0,"allocated":320630784,"data_stored":320334007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,5,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":5.3730000000000002},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.22900000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":5.577},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":5.4939999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":815700,"kb_used_data":517900,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93556140,"statfs":{"total":96636764160,"available":95801487360,"internally_reserved":0,"allocated":530329600,"data_stored":530102403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.312},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.43099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.248},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":531116032,"data_stored":531112553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":321310720,"data_stored":321237661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":432177152,"data_stored":432173673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:26.789 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:58:26.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'9678", 'reported_seq': 9745, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:22.451684+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:58:22.451684+0000', 'last_peered': '2024-09-18T11:58:22.451684+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:58:22.451684+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9678, 'log_dups_size': 0, 'ondisk_log_size': 9678, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39641088, 'num_objects': 9678, 'num_object_clones': 0, 'num_object_copies': 19356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9678, 'num_write_kb': 38712, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6079}, {'shards': '0,2', 'objects': 3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:26.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:58:26.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'9694", 'reported_seq': 9879, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:22.921115+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:58:22.921115+0000', 'last_peered': '2024-09-18T11:58:22.921115+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:58:22.921115+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 9694, 'log_dups_size': 0, 'ondisk_log_size': 9694, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39690240, 'num_objects': 9690, 'num_object_clones': 0, 'num_object_copies': 19380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6552, 'num_write_kb': 26208, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3535}, {'shards': '2', 'objects': 6154}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:26.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:58:26.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'9618", 'reported_seq': 9699, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:22.442472+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:58:22.442472+0000', 'last_peered': '2024-09-18T11:58:22.442472+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:58:22.442472+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9618, 'log_dups_size': 0, 'ondisk_log_size': 9618, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39376082, 'num_objects': 9615, 'num_object_clones': 0, 'num_object_copies': 19230, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9615, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9616, 'num_write_kb': 38455, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6088}, {'shards': '0,2', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:26.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:58:26.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'9610", 'reported_seq': 9697, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:22.339135+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:58:22.339135+0000', 'last_peered': '2024-09-18T11:58:22.339135+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:58:22.339135+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9610, 'log_dups_size': 0, 'ondisk_log_size': 9610, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39339218, 'num_objects': 9606, 'num_object_clones': 0, 'num_object_copies': 19212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9608, 'num_write_kb': 38420, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6047}, {'shards': '0,2', 'objects': 3559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:26.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:58:26.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'9654", 'reported_seq': 9732, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:22.429806+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:58:22.429806+0000', 'last_peered': '2024-09-18T11:58:22.429806+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:58:22.429806+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9654, 'log_dups_size': 0, 'ondisk_log_size': 9654, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39511250, 'num_objects': 9648, 'num_object_clones': 0, 'num_object_copies': 19296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9648, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9652, 'num_write_kb': 38588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6116}, {'shards': '0,2', 'objects': 3532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:26.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:58:26.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'9638", 'reported_seq': 9694, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:22.877509+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:58:22.877509+0000', 'last_peered': '2024-09-18T11:58:22.877509+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:58:22.877509+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9638, 'log_dups_size': 0, 'ondisk_log_size': 9638, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39458002, 'num_objects': 9635, 'num_object_clones': 0, 'num_object_copies': 19270, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9635, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9636, 'num_write_kb': 38535, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3550}, {'shards': '2', 'objects': 6085}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:26.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:58:26.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'9670", 'reported_seq': 9879, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:22.850435+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:58:22.850435+0000', 'last_peered': '2024-09-18T11:58:22.850435+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:58:22.850435+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 9670, 'log_dups_size': 0, 'ondisk_log_size': 9670, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39583744, 'num_objects': 9664, 'num_object_clones': 0, 'num_object_copies': 19328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9664, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6150, 'num_write_kb': 24600, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3592}, {'shards': '2', 'objects': 6071}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:26.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:58:26.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'9665", 'reported_seq': 9745, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:22.333444+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:58:22.333444+0000', 'last_peered': '2024-09-18T11:58:22.333444+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:58:22.333444+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9665, 'log_dups_size': 0, 'ondisk_log_size': 9665, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39580265, 'num_objects': 9664, 'num_object_clones': 0, 'num_object_copies': 19328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9664, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9664, 'num_write_kb': 38653, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6129}, {'shards': '0,2', 'objects': 3535}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:26.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:58:26.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'9652", 'reported_seq': 13467, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:58:22.478189+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:58:22.478189+0000', 'last_peered': '2024-09-18T11:58:22.478189+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:58:22.478189+0000', 'last_undegraded': '2024-09-18T11:58:22.478189+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9652, 'log_dups_size': 0, 'ondisk_log_size': 9652, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39503675, 'num_objects': 9647, 'num_object_clones': 0, 'num_object_copies': 19294, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1657, 'num_objects_unfound': 0, 'num_objects_dirty': 9647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9649, 'num_write_kb': 38580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1868, 'num_bytes_recovered': 7639040, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7990}, {'shards': '0,2', 'objects': 1657}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:26.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:58:26.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'9517", 'reported_seq': 9598, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:22.451753+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:58:22.451753+0000', 'last_peered': '2024-09-18T11:58:22.451753+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:58:22.451753+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9517, 'log_dups_size': 0, 'ondisk_log_size': 9517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38966482, 'num_objects': 9515, 'num_object_clones': 0, 'num_object_copies': 19030, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9515, 'num_write_kb': 38054, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5922}, {'shards': '0,2', 'objects': 3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:26.792 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:58:26.792 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'9664", 'reported_seq': 9749, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:22.331368+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:58:22.331368+0000', 'last_peered': '2024-09-18T11:58:22.331368+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:58:22.331368+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9664, 'log_dups_size': 0, 'ondisk_log_size': 9664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39537060, 'num_objects': 9656, 'num_object_clones': 0, 'num_object_copies': 19312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9661, 'num_write_kb': 38615, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6133}, {'shards': '0,2', 'objects': 3522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:26.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:27.759 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:27.759 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/308258 objects degraded (11.537%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":307,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":154129,"data_bytes":631234631,"bytes_used":2215297024,"bytes_avail":287694995456,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":308258,"degraded_ratio":0.11537089061759956,"misplaced_objects":1657,"misplaced_total":308258,"misplaced_ratio":0.0053753673870588926,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":85654,"recovering_keys_per_sec":0,"num_objects_recovered":251,"num_bytes_recovered":1028096,"num_keys_recovered":0,"write_bytes_sec":1398115,"read_op_per_sec":0,"write_op_per_sec":341},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:27.828 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:27.829 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:28.704 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:28.717 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:28.735 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-18T11:58:27.109946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631234631,"num_objects":154129,"num_object_clones":0,"num_object_copies":308258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1657,"num_objects_unfound":0,"num_objects_dirty":154129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138184,"num_write_kb":552610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5369,"num_bytes_recovered":21897852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154192,"ondisk_log_size":154192,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2163376,"kb_used_data":1253208,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":280952144,"statfs":{"total":289910292480,"available":287694995456,"internally_reserved":0,"allocated":1283284992,"data_stored":1282537443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,7,1,8,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16781312,"num_objects":4097,"num_object_clones":0,"num_object_copies":8194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-251,"num_objects_unfound":0,"num_objects_dirty":4097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4097,"num_write_kb":16388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":1028096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002807"},"pg_stats":[{"pgid":"2.f","version":"39'9678","reported_seq":9745,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.451684+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:22.451684+0000","last_peered":"2024-09-18T11:58:22.451684+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:22.451684+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9678,"log_dups_size":0,"ondisk_log_size":9678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39641088,"num_objects":9678,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":38712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6079},{"shards":"0,2","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9644","reported_seq":9802,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.913197+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:22.913197+0000","last_peered":"2024-09-18T11:58:22.913197+0000","last_clean":"2024-09-18T11:58:22.913197+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:22.913197+0000","last_undegraded":"2024-09-18T11:58:22.913197+0000","last_fullsized":"2024-09-18T11:58:22.913197+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39477248,"num_objects":9638,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6836,"num_write_kb":27344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9694","reported_seq":9879,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.921115+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:22.921115+0000","last_peered":"2024-09-18T11:58:22.921115+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:22.921115+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9694,"log_dups_size":0,"ondisk_log_size":9694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39690240,"num_objects":9690,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6552,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3535},{"shards":"2","objects":6154}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9538","reported_seq":9567,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.870308+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:22.870308+0000","last_peered":"2024-09-18T11:58:22.870308+0000","last_clean":"2024-09-18T11:58:22.870308+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:22.870308+0000","last_undegraded":"2024-09-18T11:58:22.870308+0000","last_fullsized":"2024-09-18T11:58:22.870308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9538,"log_dups_size":0,"ondisk_log_size":9538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39067648,"num_objects":9538,"num_object_clones":0,"num_object_copies":19076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9538,"num_write_kb":38152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9633","reported_seq":9824,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.461022+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:22.461022+0000","last_peered":"2024-09-18T11:58:22.461022+0000","last_clean":"2024-09-18T11:58:22.461022+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:22.461022+0000","last_undegraded":"2024-09-18T11:58:22.461022+0000","last_fullsized":"2024-09-18T11:58:22.461022+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9633,"log_dups_size":0,"ondisk_log_size":9633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39428096,"num_objects":9626,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6798,"num_write_kb":27192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9618","reported_seq":9699,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.442472+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:22.442472+0000","last_peered":"2024-09-18T11:58:22.442472+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:22.442472+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9618,"log_dups_size":0,"ondisk_log_size":9618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39376082,"num_objects":9615,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9616,"num_write_kb":38455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6088},{"shards":"0,2","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9610","reported_seq":9697,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.339135+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:22.339135+0000","last_peered":"2024-09-18T11:58:22.339135+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:22.339135+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9610,"log_dups_size":0,"ondisk_log_size":9610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39339218,"num_objects":9606,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9608,"num_write_kb":38420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6047},{"shards":"0,2","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9711","reported_seq":9935,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.470152+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:22.470152+0000","last_peered":"2024-09-18T11:58:22.470152+0000","last_clean":"2024-09-18T11:58:22.470152+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:22.470152+0000","last_undegraded":"2024-09-18T11:58:22.470152+0000","last_fullsized":"2024-09-18T11:58:22.470152+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9711,"log_dups_size":0,"ondisk_log_size":9711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39768064,"num_objects":9709,"num_object_clones":0,"num_object_copies":19418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6028,"num_write_kb":24112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9603","reported_seq":16731,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:22.703150+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:22.703150+0000","last_peered":"2024-09-18T11:58:22.703150+0000","last_clean":"2024-09-18T11:58:22.703150+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:22.703150+0000","last_undegraded":"2024-09-18T11:58:22.703150+0000","last_fullsized":"2024-09-18T11:58:22.703150+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39306450,"num_objects":9598,"num_object_clones":0,"num_object_copies":19196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":38388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9654","reported_seq":9732,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.429806+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:22.429806+0000","last_peered":"2024-09-18T11:58:22.429806+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:22.429806+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9654,"log_dups_size":0,"ondisk_log_size":9654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39511250,"num_objects":9648,"num_object_clones":0,"num_object_copies":19296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9652,"num_write_kb":38588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6116},{"shards":"0,2","objects":3532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9638","reported_seq":9694,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.877509+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:22.877509+0000","last_peered":"2024-09-18T11:58:22.877509+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:22.877509+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9638,"log_dups_size":0,"ondisk_log_size":9638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458002,"num_objects":9635,"num_object_clones":0,"num_object_copies":19270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9636,"num_write_kb":38535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3550},{"shards":"2","objects":6085}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9670","reported_seq":9879,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.850435+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:22.850435+0000","last_peered":"2024-09-18T11:58:22.850435+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:22.850435+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9670,"log_dups_size":0,"ondisk_log_size":9670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39583744,"num_objects":9664,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6150,"num_write_kb":24600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3592},{"shards":"2","objects":6071}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9665","reported_seq":9745,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.333444+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:22.333444+0000","last_peered":"2024-09-18T11:58:22.333444+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:22.333444+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9665,"log_dups_size":0,"ondisk_log_size":9665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39580265,"num_objects":9664,"num_object_clones":0,"num_object_copies":19328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9664,"num_write_kb":38653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6129},{"shards":"0,2","objects":3535}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9652","reported_seq":13467,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:22.478189+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:22.478189+0000","last_peered":"2024-09-18T11:58:22.478189+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:22.478189+0000","last_undegraded":"2024-09-18T11:58:22.478189+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39503675,"num_objects":9647,"num_object_clones":0,"num_object_copies":19294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1657,"num_objects_unfound":0,"num_objects_dirty":9647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9649,"num_write_kb":38580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1868,"num_bytes_recovered":7639040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7990},{"shards":"0,2","objects":1657}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9517","reported_seq":9598,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.451753+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:22.451753+0000","last_peered":"2024-09-18T11:58:22.451753+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:22.451753+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38966482,"num_objects":9515,"num_object_clones":0,"num_object_copies":19030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9515,"num_write_kb":38054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5922},{"shards":"0,2","objects":3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9664","reported_seq":9749,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:22.331368+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:22.331368+0000","last_peered":"2024-09-18T11:58:22.331368+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:22.331368+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9664,"log_dups_size":0,"ondisk_log_size":9664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39537060,"num_objects":9656,"num_object_clones":0,"num_object_copies":19312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9661,"num_write_kb":38615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6133},{"shards":"0,2","objects":3522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":631234612,"num_objects":154127,"num_object_clones":0,"num_object_copies":308254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1657,"num_objects_unfound":0,"num_objects_dirty":154127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138182,"num_write_kb":552608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5368,"num_bytes_recovered":21897833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1284603904,"data_stored":1284523887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154189,"ondisk_log_size":154189,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051638,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732536,"kb_used_data":422192,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93639304,"statfs":{"total":96636764160,"available":95886647296,"internally_reserved":0,"allocated":432324608,"data_stored":432101033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.24099999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.214},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.26800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855401,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615140,"kb_used_data":313116,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93756700,"statfs":{"total":96636764160,"available":96006860800,"internally_reserved":0,"allocated":320630784,"data_stored":320334007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,5,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":5.3730000000000002},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.22900000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":5.577},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":5.4939999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":815700,"kb_used_data":517900,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93556140,"statfs":{"total":96636764160,"available":95801487360,"internally_reserved":0,"allocated":530329600,"data_stored":530102403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.312},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.43099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:57:22 2024","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":0.80900000000000005,"15min":0.58199999999999996},"min":{"1min":0.22500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":8.6080000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.248},{"interface":"front","average":{"1min":1.208,"5min":0.59599999999999997,"15min":0.41299999999999998},"min":{"1min":0.21299999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.4309999999999992,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":531116032,"data_stored":531112553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":321310720,"data_stored":321237661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":432177152,"data_stored":432173673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:31.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:32.614 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:32.641 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:32.655 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-18T11:58:31.111502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634728543,"num_objects":154983,"num_object_clones":0,"num_object_copies":309966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1537,"num_objects_unfound":0,"num_objects_dirty":154983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140236,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5489,"num_bytes_recovered":22389372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156244,"ondisk_log_size":156244,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2175552,"kb_used_data":1265380,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":280939968,"statfs":{"total":289910292480,"available":287682527232,"internally_reserved":0,"allocated":1295749120,"data_stored":1294993427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,10,6,4,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12001304,"num_objects":2931,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-235,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4129,"num_write_kb":14117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":962560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003495"},"pg_stats":[{"pgid":"2.f","version":"39'9800","reported_seq":9871,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:27.486186+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:27.486186+0000","last_peered":"2024-09-18T11:58:27.486186+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:27.486186+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9800,"log_dups_size":0,"ondisk_log_size":9800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39878656,"num_objects":9736,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9800,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6137},{"shards":"0,2","objects":3567}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9764","reported_seq":9922,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.798235+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:27.798235+0000","last_peered":"2024-09-18T11:58:27.798235+0000","last_clean":"2024-09-18T11:58:27.798235+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:27.798235+0000","last_undegraded":"2024-09-18T11:58:27.798235+0000","last_fullsized":"2024-09-18T11:58:27.798235+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9764,"log_dups_size":0,"ondisk_log_size":9764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39600128,"num_objects":9668,"num_object_clones":0,"num_object_copies":19336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6956,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9818","reported_seq":10004,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:27.840771+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:27.840771+0000","last_peered":"2024-09-18T11:58:27.840771+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:27.840771+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9818,"log_dups_size":0,"ondisk_log_size":9818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39890968,"num_objects":9740,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9740,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6676,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3498},{"shards":"2","objects":6204}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9677","reported_seq":9706,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.873200+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:27.873200+0000","last_peered":"2024-09-18T11:58:27.873200+0000","last_clean":"2024-09-18T11:58:27.873200+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:27.873200+0000","last_undegraded":"2024-09-18T11:58:27.873200+0000","last_fullsized":"2024-09-18T11:58:27.873200+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9677,"log_dups_size":0,"ondisk_log_size":9677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39251968,"num_objects":9583,"num_object_clones":0,"num_object_copies":19166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9677,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9749","reported_seq":9940,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.397208+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:27.397208+0000","last_peered":"2024-09-18T11:58:27.397208+0000","last_clean":"2024-09-18T11:58:27.397208+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:27.397208+0000","last_undegraded":"2024-09-18T11:58:27.397208+0000","last_fullsized":"2024-09-18T11:58:27.397208+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9749,"log_dups_size":0,"ondisk_log_size":9749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39723008,"num_objects":9698,"num_object_clones":0,"num_object_copies":19396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6914,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9758","reported_seq":9839,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:27.361442+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:27.361442+0000","last_peered":"2024-09-18T11:58:27.361442+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:27.361442+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9758,"log_dups_size":0,"ondisk_log_size":9758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39589074,"num_objects":9667,"num_object_clones":0,"num_object_copies":19334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9756,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6140},{"shards":"0,2","objects":3483}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9711","reported_seq":9798,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:27.426508+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:27.426508+0000","last_peered":"2024-09-18T11:58:27.426508+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:27.426508+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9711,"log_dups_size":0,"ondisk_log_size":9711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39572690,"num_objects":9663,"num_object_clones":0,"num_object_copies":19326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9709,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6104},{"shards":"0,2","objects":3537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9835","reported_seq":10059,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.463389+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:27.463389+0000","last_peered":"2024-09-18T11:58:27.463389+0000","last_clean":"2024-09-18T11:58:27.463389+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:27.463389+0000","last_undegraded":"2024-09-18T11:58:27.463389+0000","last_fullsized":"2024-09-18T11:58:27.463389+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9835,"log_dups_size":0,"ondisk_log_size":9835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40005632,"num_objects":9767,"num_object_clones":0,"num_object_copies":19534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6152,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9746","reported_seq":16873,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:27.800775+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:27.800775+0000","last_peered":"2024-09-18T11:58:27.800775+0000","last_clean":"2024-09-18T11:58:27.800775+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:27.800775+0000","last_undegraded":"2024-09-18T11:58:27.800775+0000","last_fullsized":"2024-09-18T11:58:27.800775+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39564498,"num_objects":9661,"num_object_clones":0,"num_object_copies":19322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9751","reported_seq":9829,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:27.321901+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:27.321901+0000","last_peered":"2024-09-18T11:58:27.321901+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:27.321901+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9751,"log_dups_size":0,"ondisk_log_size":9751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39711954,"num_objects":9697,"num_object_clones":0,"num_object_copies":19394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9749,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6165},{"shards":"0,2","objects":3508}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9779","reported_seq":9835,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:27.919234+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:27.919234+0000","last_peered":"2024-09-18T11:58:27.919234+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:27.919234+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9779,"log_dups_size":0,"ondisk_log_size":9779,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39666898,"num_objects":9686,"num_object_clones":0,"num_object_copies":19372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9777,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3505},{"shards":"2","objects":6136}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9803","reported_seq":10012,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:27.872964+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:27.872964+0000","last_peered":"2024-09-18T11:58:27.872964+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:27.872964+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9803,"log_dups_size":0,"ondisk_log_size":9803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39710720,"num_objects":9695,"num_object_clones":0,"num_object_copies":19390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6283,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3541},{"shards":"2","objects":6102}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9812","reported_seq":9894,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:27.486283+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:27.486283+0000","last_peered":"2024-09-18T11:58:27.486283+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:27.486283+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9812,"log_dups_size":0,"ondisk_log_size":9812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39797353,"num_objects":9717,"num_object_clones":0,"num_object_copies":19434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6182},{"shards":"0,2","objects":3488}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9800","reported_seq":13854,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:27.505948+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:27.505948+0000","last_peered":"2024-09-18T11:58:27.505948+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:27.505948+0000","last_undegraded":"2024-09-18T11:58:27.505948+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9800,"log_dups_size":0,"ondisk_log_size":9800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39823163,"num_objects":9725,"num_object_clones":0,"num_object_copies":19450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1537,"num_objects_unfound":0,"num_objects_dirty":9725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9797,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1988,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8188},{"shards":"0,2","objects":1537}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9645","reported_seq":9725,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:27.428726+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:27.428726+0000","last_peered":"2024-09-18T11:58:27.428726+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:27.428726+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9645,"log_dups_size":0,"ondisk_log_size":9645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39130322,"num_objects":9555,"num_object_clones":0,"num_object_copies":19110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9643,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5962},{"shards":"0,2","objects":3549}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9793","reported_seq":9878,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:27.407571+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:27.407571+0000","last_peered":"2024-09-18T11:58:27.407571+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:27.407571+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9793,"log_dups_size":0,"ondisk_log_size":9793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39811492,"num_objects":9723,"num_object_clones":0,"num_object_copies":19446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6200},{"shards":"0,2","objects":3491}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":634728524,"num_objects":154981,"num_object_clones":0,"num_object_copies":309962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1537,"num_objects_unfound":0,"num_objects_dirty":154981,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140234,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5488,"num_bytes_recovered":22389353,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1293365248,"data_stored":1293280566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156241,"ondisk_log_size":156241,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051639,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721732,"kb_used_data":423868,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93650108,"statfs":{"total":96636764160,"available":95897710592,"internally_reserved":0,"allocated":434040832,"data_stored":433813185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.32800000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":2.4289999999999998},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.48099999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855402,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624824,"kb_used_data":318640,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93747016,"statfs":{"total":96636764160,"available":95996944384,"internally_reserved":0,"allocated":326287360,"data_stored":325986511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":1.6619999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.32200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247866,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":828996,"kb_used_data":522872,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93542844,"statfs":{"total":96636764160,"available":95787872256,"internally_reserved":0,"allocated":535420928,"data_stored":535193731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,9,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":7.0970000000000004},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":7.0339999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":533913600,"data_stored":533910121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":325873664,"data_stored":325800012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":433577984,"data_stored":433570433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:32.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:33.557 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:33.599 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:33.628 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-18T11:58:33.111942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":625963103,"num_objects":152843,"num_object_clones":0,"num_object_copies":305686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1390,"num_objects_unfound":0,"num_objects_dirty":152843,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142376,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5639,"num_bytes_recovered":22991484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158384,"ondisk_log_size":158384,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2175996,"kb_used_data":1253348,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":280939524,"statfs":{"total":289910292480,"available":287682072576,"internally_reserved":0,"allocated":1283428352,"data_stored":1282683096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,3,6,8,13,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3235864,"num_objects":791,"num_object_clones":0,"num_object_copies":1582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-382,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6269,"num_write_kb":14117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":1564672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003360"},"pg_stats":[{"pgid":"2.f","version":"39'9894","reported_seq":9965,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.554946+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:32.554946+0000","last_peered":"2024-09-18T11:58:32.554946+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:32.554946+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39493632,"num_objects":9642,"num_object_clones":0,"num_object_copies":19284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9894,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6043},{"shards":"0,2","objects":3473}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9901","reported_seq":10059,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:32.747455+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:32.747455+0000","last_peered":"2024-09-18T11:58:32.747455+0000","last_clean":"2024-09-18T11:58:32.747455+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:32.747455+0000","last_undegraded":"2024-09-18T11:58:32.747455+0000","last_fullsized":"2024-09-18T11:58:32.747455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9901,"log_dups_size":0,"ondisk_log_size":9901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39038976,"num_objects":9531,"num_object_clones":0,"num_object_copies":19062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7093,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9956","reported_seq":10142,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.947214+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:32.947214+0000","last_peered":"2024-09-18T11:58:32.947214+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:32.947214+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9956,"log_dups_size":0,"ondisk_log_size":9956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39325720,"num_objects":9602,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9602,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6814,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3360},{"shards":"2","objects":6066}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9811","reported_seq":9839,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:32.950276+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:32.950276+0000","last_peered":"2024-09-18T11:58:32.950276+0000","last_clean":"2024-09-18T11:58:32.950276+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:32.950276+0000","last_undegraded":"2024-09-18T11:58:32.950276+0000","last_fullsized":"2024-09-18T11:58:32.950276+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9811,"log_dups_size":0,"ondisk_log_size":9811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38703104,"num_objects":9449,"num_object_clones":0,"num_object_copies":18898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9878","reported_seq":10069,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:32.529073+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:32.529073+0000","last_peered":"2024-09-18T11:58:32.529073+0000","last_clean":"2024-09-18T11:58:32.529073+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:32.529073+0000","last_undegraded":"2024-09-18T11:58:32.529073+0000","last_fullsized":"2024-09-18T11:58:32.529073+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9878,"log_dups_size":0,"ondisk_log_size":9878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39194624,"num_objects":9569,"num_object_clones":0,"num_object_copies":19138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7043,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9912","reported_seq":9991,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.433231+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:32.433231+0000","last_peered":"2024-09-18T11:58:32.433231+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:32.433231+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9912,"log_dups_size":0,"ondisk_log_size":9912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38958290,"num_objects":9513,"num_object_clones":0,"num_object_copies":19026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9910,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5986},{"shards":"0,2","objects":3329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9828","reported_seq":9915,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.464648+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:32.464648+0000","last_peered":"2024-09-18T11:58:32.464648+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:32.464648+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9828,"log_dups_size":0,"ondisk_log_size":9828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39093458,"num_objects":9546,"num_object_clones":0,"num_object_copies":19092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9826,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5987},{"shards":"0,2","objects":3420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9960","reported_seq":10185,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:32.507903+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:32.507903+0000","last_peered":"2024-09-18T11:58:32.507903+0000","last_clean":"2024-09-18T11:58:32.507903+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:32.507903+0000","last_undegraded":"2024-09-18T11:58:32.507903+0000","last_fullsized":"2024-09-18T11:58:32.507903+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9960,"log_dups_size":0,"ondisk_log_size":9960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39493632,"num_objects":9642,"num_object_clones":0,"num_object_copies":19284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6277,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9897","reported_seq":17025,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:32.794192+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:32.794192+0000","last_peered":"2024-09-18T11:58:32.794192+0000","last_clean":"2024-09-18T11:58:32.794192+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:32.794192+0000","last_undegraded":"2024-09-18T11:58:32.794192+0000","last_fullsized":"2024-09-18T11:58:32.794192+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38946002,"num_objects":9510,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9895,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9889","reported_seq":9966,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.534828+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:32.534828+0000","last_peered":"2024-09-18T11:58:32.534828+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:32.534828+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9889,"log_dups_size":0,"ondisk_log_size":9889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39146706,"num_objects":9559,"num_object_clones":0,"num_object_copies":19118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9887,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6027},{"shards":"0,2","objects":3370}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9922","reported_seq":9977,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.978087+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:32.978087+0000","last_peered":"2024-09-18T11:58:32.978087+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:32.978087+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9922,"log_dups_size":0,"ondisk_log_size":9922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39081170,"num_objects":9543,"num_object_clones":0,"num_object_copies":19086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9920,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3362},{"shards":"2","objects":5993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9941","reported_seq":10148,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.918333+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:32.918333+0000","last_peered":"2024-09-18T11:58:32.918333+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:32.918333+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9941,"log_dups_size":0,"ondisk_log_size":9941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39145472,"num_objects":9557,"num_object_clones":0,"num_object_copies":19114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6421,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3403},{"shards":"2","objects":5964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9951","reported_seq":10033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.547239+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:32.547239+0000","last_peered":"2024-09-18T11:58:32.547239+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:32.547239+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9951,"log_dups_size":0,"ondisk_log_size":9951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39228009,"num_objects":9578,"num_object_clones":0,"num_object_copies":19156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6043},{"shards":"0,2","objects":3349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9915","reported_seq":14270,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:32.525305+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:32.525305+0000","last_peered":"2024-09-18T11:58:32.525305+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:32.525305+0000","last_undegraded":"2024-09-18T11:58:32.525305+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9915,"log_dups_size":0,"ondisk_log_size":9915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39352123,"num_objects":9610,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1390,"num_objects_unfound":0,"num_objects_dirty":9610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9912,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":8732672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8220},{"shards":"0,2","objects":1390}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9785","reported_seq":9866,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.472610+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:32.472610+0000","last_peered":"2024-09-18T11:58:32.472610+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:32.472610+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9785,"log_dups_size":0,"ondisk_log_size":9785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38556882,"num_objects":9415,"num_object_clones":0,"num_object_copies":18830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9783,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5822},{"shards":"0,2","objects":3409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9941","reported_seq":10026,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.458326+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:32.458326+0000","last_peered":"2024-09-18T11:58:32.458326+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:32.458326+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9941,"log_dups_size":0,"ondisk_log_size":9941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39205284,"num_objects":9575,"num_object_clones":0,"num_object_copies":19150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9938,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6052},{"shards":"0,2","objects":3343}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":625963084,"num_objects":152841,"num_object_clones":0,"num_object_copies":305682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1390,"num_objects_unfound":0,"num_objects_dirty":152841,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142374,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5638,"num_bytes_recovered":22991465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1281945600,"data_stored":1281867876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158381,"ondisk_log_size":158381,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051640,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721604,"kb_used_data":419580,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93650236,"statfs":{"total":96636764160,"available":95897841664,"internally_reserved":0,"allocated":429649920,"data_stored":429422273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.624},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.95099999999999996}]}]},{"osd":1,"up_from":35,"seq":150323855403,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628464,"kb_used_data":318120,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93743376,"statfs":{"total":96636764160,"available":95993217024,"internally_reserved":0,"allocated":325754880,"data_stored":325464468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.246},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":3.0739999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.27400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":825928,"kb_used_data":515648,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93545912,"statfs":{"total":96636764160,"available":95791013888,"internally_reserved":0,"allocated":528023552,"data_stored":527796355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,6,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.313}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":527478784,"data_stored":527475305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":325361664,"data_stored":325294970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":429105152,"data_stored":429097601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:33.630 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:58:33.630 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'9894", 'reported_seq': 9965, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:32.554946+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:58:32.554946+0000', 'last_peered': '2024-09-18T11:58:32.554946+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:58:32.554946+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9894, 'log_dups_size': 0, 'ondisk_log_size': 9894, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39493632, 'num_objects': 9642, 'num_object_clones': 0, 'num_object_copies': 19284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9894, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6043}, {'shards': '0,2', 'objects': 3473}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:33.630 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:58:33.630 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'9956", 'reported_seq': 10142, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:32.947214+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:58:32.947214+0000', 'last_peered': '2024-09-18T11:58:32.947214+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:58:32.947214+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 9956, 'log_dups_size': 0, 'ondisk_log_size': 9956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39325720, 'num_objects': 9602, 'num_object_clones': 0, 'num_object_copies': 19204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9602, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6814, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3360}, {'shards': '2', 'objects': 6066}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:33.630 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:58:33.630 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'9912", 'reported_seq': 9991, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:32.433231+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:58:32.433231+0000', 'last_peered': '2024-09-18T11:58:32.433231+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:58:32.433231+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9912, 'log_dups_size': 0, 'ondisk_log_size': 9912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38958290, 'num_objects': 9513, 'num_object_clones': 0, 'num_object_copies': 19026, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9910, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5986}, {'shards': '0,2', 'objects': 3329}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:33.630 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:58:33.630 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'9828", 'reported_seq': 9915, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:32.464648+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:58:32.464648+0000', 'last_peered': '2024-09-18T11:58:32.464648+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:58:32.464648+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9828, 'log_dups_size': 0, 'ondisk_log_size': 9828, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39093458, 'num_objects': 9546, 'num_object_clones': 0, 'num_object_copies': 19092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9826, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5987}, {'shards': '0,2', 'objects': 3420}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:33.630 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:58:33.631 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'9889", 'reported_seq': 9966, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:32.534828+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:58:32.534828+0000', 'last_peered': '2024-09-18T11:58:32.534828+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:58:32.534828+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9889, 'log_dups_size': 0, 'ondisk_log_size': 9889, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39146706, 'num_objects': 9559, 'num_object_clones': 0, 'num_object_copies': 19118, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9887, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6027}, {'shards': '0,2', 'objects': 3370}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:33.631 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:58:33.631 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'9922", 'reported_seq': 9977, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:32.978087+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:58:32.978087+0000', 'last_peered': '2024-09-18T11:58:32.978087+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:58:32.978087+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9922, 'log_dups_size': 0, 'ondisk_log_size': 9922, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39081170, 'num_objects': 9543, 'num_object_clones': 0, 'num_object_copies': 19086, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9920, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3362}, {'shards': '2', 'objects': 5993}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:33.631 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:58:33.631 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'9941", 'reported_seq': 10148, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:32.918333+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:58:32.918333+0000', 'last_peered': '2024-09-18T11:58:32.918333+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:58:32.918333+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 9941, 'log_dups_size': 0, 'ondisk_log_size': 9941, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39145472, 'num_objects': 9557, 'num_object_clones': 0, 'num_object_copies': 19114, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6421, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3403}, {'shards': '2', 'objects': 5964}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:33.631 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:58:33.631 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'9951", 'reported_seq': 10033, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:32.547239+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:58:32.547239+0000', 'last_peered': '2024-09-18T11:58:32.547239+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:58:32.547239+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9951, 'log_dups_size': 0, 'ondisk_log_size': 9951, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39228009, 'num_objects': 9578, 'num_object_clones': 0, 'num_object_copies': 19156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9950, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6043}, {'shards': '0,2', 'objects': 3349}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:33.632 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:58:33.632 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'9915", 'reported_seq': 14270, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:58:32.525305+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:58:32.525305+0000', 'last_peered': '2024-09-18T11:58:32.525305+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:58:32.525305+0000', 'last_undegraded': '2024-09-18T11:58:32.525305+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9915, 'log_dups_size': 0, 'ondisk_log_size': 9915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39352123, 'num_objects': 9610, 'num_object_clones': 0, 'num_object_copies': 19220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1390, 'num_objects_unfound': 0, 'num_objects_dirty': 9610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9912, 'num_write_kb': 39032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2138, 'num_bytes_recovered': 8732672, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 8220}, {'shards': '0,2', 'objects': 1390}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:33.632 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:58:33.632 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'9785", 'reported_seq': 9866, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:32.472610+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:58:32.472610+0000', 'last_peered': '2024-09-18T11:58:32.472610+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:58:32.472610+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9785, 'log_dups_size': 0, 'ondisk_log_size': 9785, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38556882, 'num_objects': 9415, 'num_object_clones': 0, 'num_object_copies': 18830, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9415, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9783, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5822}, {'shards': '0,2', 'objects': 3409}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:33.632 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:58:33.632 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'9941", 'reported_seq': 10026, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:32.458326+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:58:32.458326+0000', 'last_peered': '2024-09-18T11:58:32.458326+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:58:32.458326+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9941, 'log_dups_size': 0, 'ondisk_log_size': 9941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39205284, 'num_objects': 9575, 'num_object_clones': 0, 'num_object_copies': 19150, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9575, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9938, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6052}, {'shards': '0,2', 'objects': 3343}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:33.632 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:34.740 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:34.741 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/305686 objects degraded (11.634%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":314,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":152843,"data_bytes":625963103,"bytes_used":2228219904,"bytes_avail":287682072576,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":305686,"degraded_ratio":0.11634160543826018,"misplaced_objects":1390,"misplaced_total":305686,"misplaced_ratio":0.0045471496895507155,"recovering_objects_per_sec":32,"recovering_bytes_per_sec":130352,"recovering_keys_per_sec":0,"num_objects_recovered":385,"num_bytes_recovered":1564672,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":1204313,"read_op_per_sec":0,"write_op_per_sec":522},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:34.775 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:34.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:35.621 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:35.641 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:35.656 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-18T11:58:35.114108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":625963103,"num_objects":152843,"num_object_clones":0,"num_object_copies":305686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1390,"num_objects_unfound":0,"num_objects_dirty":152843,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142376,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5639,"num_bytes_recovered":22991484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158384,"ondisk_log_size":158384,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2175996,"kb_used_data":1253348,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":280939524,"statfs":{"total":289910292480,"available":287682072576,"internally_reserved":0,"allocated":1283428352,"data_stored":1282683096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,3,6,8,13,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5271528,"num_objects":-1286,"num_object_clones":0,"num_object_copies":-2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-267,"num_objects_unfound":0,"num_objects_dirty":-1286,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4192,"num_write_kb":5809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":1093632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005100"},"pg_stats":[{"pgid":"2.f","version":"39'9894","reported_seq":9965,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.554946+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:32.554946+0000","last_peered":"2024-09-18T11:58:32.554946+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:32.554946+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39493632,"num_objects":9642,"num_object_clones":0,"num_object_copies":19284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9894,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6043},{"shards":"0,2","objects":3473}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'9901","reported_seq":10059,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:32.747455+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:32.747455+0000","last_peered":"2024-09-18T11:58:32.747455+0000","last_clean":"2024-09-18T11:58:32.747455+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:32.747455+0000","last_undegraded":"2024-09-18T11:58:32.747455+0000","last_fullsized":"2024-09-18T11:58:32.747455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":9901,"log_dups_size":0,"ondisk_log_size":9901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39038976,"num_objects":9531,"num_object_clones":0,"num_object_copies":19062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7093,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'9956","reported_seq":10142,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.947214+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:32.947214+0000","last_peered":"2024-09-18T11:58:32.947214+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:32.947214+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":9956,"log_dups_size":0,"ondisk_log_size":9956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39325720,"num_objects":9602,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9602,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6814,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3360},{"shards":"2","objects":6066}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9811","reported_seq":9839,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:32.950276+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:32.950276+0000","last_peered":"2024-09-18T11:58:32.950276+0000","last_clean":"2024-09-18T11:58:32.950276+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:32.950276+0000","last_undegraded":"2024-09-18T11:58:32.950276+0000","last_fullsized":"2024-09-18T11:58:32.950276+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9811,"log_dups_size":0,"ondisk_log_size":9811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38703104,"num_objects":9449,"num_object_clones":0,"num_object_copies":18898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9811,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'9878","reported_seq":10069,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:32.529073+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:32.529073+0000","last_peered":"2024-09-18T11:58:32.529073+0000","last_clean":"2024-09-18T11:58:32.529073+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:32.529073+0000","last_undegraded":"2024-09-18T11:58:32.529073+0000","last_fullsized":"2024-09-18T11:58:32.529073+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":9878,"log_dups_size":0,"ondisk_log_size":9878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39194624,"num_objects":9569,"num_object_clones":0,"num_object_copies":19138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7043,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'9912","reported_seq":9991,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.433231+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:32.433231+0000","last_peered":"2024-09-18T11:58:32.433231+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:32.433231+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9912,"log_dups_size":0,"ondisk_log_size":9912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38958290,"num_objects":9513,"num_object_clones":0,"num_object_copies":19026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9910,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5986},{"shards":"0,2","objects":3329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9828","reported_seq":9915,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.464648+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:32.464648+0000","last_peered":"2024-09-18T11:58:32.464648+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:32.464648+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9828,"log_dups_size":0,"ondisk_log_size":9828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39093458,"num_objects":9546,"num_object_clones":0,"num_object_copies":19092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9826,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5987},{"shards":"0,2","objects":3420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'9960","reported_seq":10185,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:32.507903+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:32.507903+0000","last_peered":"2024-09-18T11:58:32.507903+0000","last_clean":"2024-09-18T11:58:32.507903+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:32.507903+0000","last_undegraded":"2024-09-18T11:58:32.507903+0000","last_fullsized":"2024-09-18T11:58:32.507903+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":9960,"log_dups_size":0,"ondisk_log_size":9960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39493632,"num_objects":9642,"num_object_clones":0,"num_object_copies":19284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6277,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'9897","reported_seq":17025,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:32.794192+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:32.794192+0000","last_peered":"2024-09-18T11:58:32.794192+0000","last_clean":"2024-09-18T11:58:32.794192+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:32.794192+0000","last_undegraded":"2024-09-18T11:58:32.794192+0000","last_fullsized":"2024-09-18T11:58:32.794192+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38946002,"num_objects":9510,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9895,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'9889","reported_seq":9966,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.534828+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:32.534828+0000","last_peered":"2024-09-18T11:58:32.534828+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:32.534828+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9889,"log_dups_size":0,"ondisk_log_size":9889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39146706,"num_objects":9559,"num_object_clones":0,"num_object_copies":19118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9887,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6027},{"shards":"0,2","objects":3370}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'9922","reported_seq":9977,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.978087+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:32.978087+0000","last_peered":"2024-09-18T11:58:32.978087+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:32.978087+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9922,"log_dups_size":0,"ondisk_log_size":9922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39081170,"num_objects":9543,"num_object_clones":0,"num_object_copies":19086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9920,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3362},{"shards":"2","objects":5993}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'9941","reported_seq":10148,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.918333+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:32.918333+0000","last_peered":"2024-09-18T11:58:32.918333+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:32.918333+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":9941,"log_dups_size":0,"ondisk_log_size":9941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39145472,"num_objects":9557,"num_object_clones":0,"num_object_copies":19114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6421,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3403},{"shards":"2","objects":5964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'9951","reported_seq":10033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.547239+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:32.547239+0000","last_peered":"2024-09-18T11:58:32.547239+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:32.547239+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9951,"log_dups_size":0,"ondisk_log_size":9951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39228009,"num_objects":9578,"num_object_clones":0,"num_object_copies":19156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6043},{"shards":"0,2","objects":3349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'9915","reported_seq":14270,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:32.525305+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:32.525305+0000","last_peered":"2024-09-18T11:58:32.525305+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:32.525305+0000","last_undegraded":"2024-09-18T11:58:32.525305+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9915,"log_dups_size":0,"ondisk_log_size":9915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39352123,"num_objects":9610,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1390,"num_objects_unfound":0,"num_objects_dirty":9610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9912,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":8732672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8220},{"shards":"0,2","objects":1390}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9785","reported_seq":9866,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.472610+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:32.472610+0000","last_peered":"2024-09-18T11:58:32.472610+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:32.472610+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9785,"log_dups_size":0,"ondisk_log_size":9785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38556882,"num_objects":9415,"num_object_clones":0,"num_object_copies":18830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9783,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5822},{"shards":"0,2","objects":3409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'9941","reported_seq":10026,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:32.458326+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:32.458326+0000","last_peered":"2024-09-18T11:58:32.458326+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:32.458326+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9941,"log_dups_size":0,"ondisk_log_size":9941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39205284,"num_objects":9575,"num_object_clones":0,"num_object_copies":19150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9938,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6052},{"shards":"0,2","objects":3343}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":625963084,"num_objects":152841,"num_object_clones":0,"num_object_copies":305682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1390,"num_objects_unfound":0,"num_objects_dirty":152841,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142374,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5638,"num_bytes_recovered":22991465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1281945600,"data_stored":1281867876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158381,"ondisk_log_size":158381,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051640,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721604,"kb_used_data":419580,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93650236,"statfs":{"total":96636764160,"available":95897841664,"internally_reserved":0,"allocated":429649920,"data_stored":429422273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.624},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.95099999999999996}]}]},{"osd":1,"up_from":35,"seq":150323855403,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628464,"kb_used_data":318120,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93743376,"statfs":{"total":96636764160,"available":95993217024,"internally_reserved":0,"allocated":325754880,"data_stored":325464468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.246},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":3.0739999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.27400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":825928,"kb_used_data":515648,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93545912,"statfs":{"total":96636764160,"available":95791013888,"internally_reserved":0,"allocated":528023552,"data_stored":527796355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,6,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.313}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":527478784,"data_stored":527475305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":325361664,"data_stored":325294970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":429105152,"data_stored":429097601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:38.659 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:39.532 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:39.539 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:39.559 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-18T11:58:39.114946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617443423,"num_objects":150763,"num_object_clones":0,"num_object_copies":301526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1243,"num_objects_unfound":0,"num_objects_dirty":150763,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144456,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5786,"num_bytes_recovered":23593596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160464,"ondisk_log_size":160464,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2148656,"kb_used_data":1242636,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":280966864,"statfs":{"total":289910292480,"available":287710068736,"internally_reserved":0,"allocated":1272459264,"data_stored":1271714008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,5,5,11,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13791208,"num_objects":-3366,"num_object_clones":0,"num_object_copies":-6732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-414,"num_objects_unfound":0,"num_objects_dirty":-3366,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6272,"num_write_kb":5809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":1695744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005000"},"pg_stats":[{"pgid":"2.f","version":"39'10038","reported_seq":10109,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.489925+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:37.489925+0000","last_peered":"2024-09-18T11:58:37.489925+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:37.489925+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38903808,"num_objects":9498,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10038,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5899},{"shards":"0,2","objects":3329}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10022","reported_seq":10178,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.960060+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:37.960060+0000","last_peered":"2024-09-18T11:58:37.960060+0000","last_clean":"2024-09-18T11:58:37.960060+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:37.960060+0000","last_undegraded":"2024-09-18T11:58:37.960060+0000","last_fullsized":"2024-09-18T11:58:37.960060+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38543360,"num_objects":9410,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7214,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10089","reported_seq":10275,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.937728+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:37.937728+0000","last_peered":"2024-09-18T11:58:37.937728+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:37.937728+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38780952,"num_objects":9469,"num_object_clones":0,"num_object_copies":18938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9469,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6947,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3227},{"shards":"2","objects":5933}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9952","reported_seq":9981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.973480+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:37.973480+0000","last_peered":"2024-09-18T11:58:37.973480+0000","last_clean":"2024-09-18T11:58:37.973480+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:37.973480+0000","last_undegraded":"2024-09-18T11:58:37.973480+0000","last_fullsized":"2024-09-18T11:58:37.973480+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9952,"log_dups_size":0,"ondisk_log_size":9952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38125568,"num_objects":9308,"num_object_clones":0,"num_object_copies":18616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9952,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10025","reported_seq":10216,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.558516+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:37.558516+0000","last_peered":"2024-09-18T11:58:37.558516+0000","last_clean":"2024-09-18T11:58:37.558516+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:37.558516+0000","last_undegraded":"2024-09-18T11:58:37.558516+0000","last_fullsized":"2024-09-18T11:58:37.558516+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38592512,"num_objects":9422,"num_object_clones":0,"num_object_copies":18844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7190,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10035","reported_seq":10116,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.551934+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:37.551934+0000","last_peered":"2024-09-18T11:58:37.551934+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:37.551934+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38454482,"num_objects":9390,"num_object_clones":0,"num_object_copies":18780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10033,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5863},{"shards":"0,2","objects":3206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9949","reported_seq":10036,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.379815+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:37.379815+0000","last_peered":"2024-09-18T11:58:37.379815+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:37.379815+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38597842,"num_objects":9425,"num_object_clones":0,"num_object_copies":18850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9947,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5866},{"shards":"0,2","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10097","reported_seq":10321,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.517675+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:37.517675+0000","last_peered":"2024-09-18T11:58:37.517675+0000","last_clean":"2024-09-18T11:58:37.517675+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:37.517675+0000","last_undegraded":"2024-09-18T11:58:37.517675+0000","last_fullsized":"2024-09-18T11:58:37.517675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38932480,"num_objects":9505,"num_object_clones":0,"num_object_copies":19010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6414,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10034","reported_seq":17161,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.826742+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:37.826742+0000","last_peered":"2024-09-18T11:58:37.826742+0000","last_clean":"2024-09-18T11:58:37.826742+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:37.826742+0000","last_undegraded":"2024-09-18T11:58:37.826742+0000","last_fullsized":"2024-09-18T11:58:37.826742+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38384850,"num_objects":9373,"num_object_clones":0,"num_object_copies":18746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10032,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10017","reported_seq":10095,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.428204+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:37.428204+0000","last_peered":"2024-09-18T11:58:37.428204+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:37.428204+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38622418,"num_objects":9431,"num_object_clones":0,"num_object_copies":18862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10015,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5899},{"shards":"0,2","objects":3242}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10044","reported_seq":10100,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.946306+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:37.946306+0000","last_peered":"2024-09-18T11:58:37.946306+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:37.946306+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38581458,"num_objects":9421,"num_object_clones":0,"num_object_copies":18842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10042,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3240},{"shards":"2","objects":5871}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10073","reported_seq":10282,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.944223+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:37.944223+0000","last_peered":"2024-09-18T11:58:37.944223+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:37.944223+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38604800,"num_objects":9425,"num_object_clones":0,"num_object_copies":18850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3271},{"shards":"2","objects":5832}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10075","reported_seq":10157,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.483720+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:37.483720+0000","last_peered":"2024-09-18T11:58:37.483720+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:37.483720+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38720105,"num_objects":9454,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5919},{"shards":"0,2","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10050","reported_seq":14698,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:37.544497+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:37.544497+0000","last_peered":"2024-09-18T11:58:37.544497+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:37.544497+0000","last_undegraded":"2024-09-18T11:58:37.544497+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38799163,"num_objects":9475,"num_object_clones":0,"num_object_copies":18950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1243,"num_objects_unfound":0,"num_objects_dirty":9475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10047,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2285,"num_bytes_recovered":9334784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8232},{"shards":"0,2","objects":1243}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9908","reported_seq":9989,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.390088+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:37.390088+0000","last_peered":"2024-09-18T11:58:37.390088+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:37.390088+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9908,"log_dups_size":0,"ondisk_log_size":9908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38053074,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9906,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5699},{"shards":"0,2","objects":3286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10053","reported_seq":10138,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.494647+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:37.494647+0000","last_peered":"2024-09-18T11:58:37.494647+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:37.494647+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38746532,"num_objects":9463,"num_object_clones":0,"num_object_copies":18926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5940},{"shards":"0,2","objects":3231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617443404,"num_objects":150761,"num_object_clones":0,"num_object_copies":301522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1243,"num_objects_unfound":0,"num_objects_dirty":150761,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144454,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5785,"num_bytes_recovered":23593577,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1270697984,"data_stored":1270620260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160461,"ondisk_log_size":160461,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051641,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721628,"kb_used_data":415436,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93650212,"statfs":{"total":96636764160,"available":95897817088,"internally_reserved":0,"allocated":425406464,"data_stored":425178817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.41799999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":11.109}]}]},{"osd":1,"up_from":35,"seq":150323855404,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619648,"kb_used_data":317624,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93752192,"statfs":{"total":96636764160,"available":96002244608,"internally_reserved":0,"allocated":325246976,"data_stored":324956564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.193},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.27100000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.22500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247868,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":807380,"kb_used_data":509576,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93564460,"statfs":{"total":96636764160,"available":95810007040,"internally_reserved":0,"allocated":521805824,"data_stored":521578627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,3,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.23899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":521129984,"data_stored":521126505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324820992,"data_stored":324754298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":424747008,"data_stored":424739457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:39.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:40.367 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:40.391 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:40.419 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-18T11:58:39.114946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617443423,"num_objects":150763,"num_object_clones":0,"num_object_copies":301526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1243,"num_objects_unfound":0,"num_objects_dirty":150763,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144456,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5786,"num_bytes_recovered":23593596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160464,"ondisk_log_size":160464,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2148656,"kb_used_data":1242636,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":280966864,"statfs":{"total":289910292480,"available":287710068736,"internally_reserved":0,"allocated":1272459264,"data_stored":1271714008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,5,5,11,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13791208,"num_objects":-3366,"num_object_clones":0,"num_object_copies":-6732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-414,"num_objects_unfound":0,"num_objects_dirty":-3366,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6272,"num_write_kb":5809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":1695744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005000"},"pg_stats":[{"pgid":"2.f","version":"39'10038","reported_seq":10109,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.489925+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:37.489925+0000","last_peered":"2024-09-18T11:58:37.489925+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:37.489925+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38903808,"num_objects":9498,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10038,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5899},{"shards":"0,2","objects":3329}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10022","reported_seq":10178,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.960060+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:37.960060+0000","last_peered":"2024-09-18T11:58:37.960060+0000","last_clean":"2024-09-18T11:58:37.960060+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:37.960060+0000","last_undegraded":"2024-09-18T11:58:37.960060+0000","last_fullsized":"2024-09-18T11:58:37.960060+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38543360,"num_objects":9410,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7214,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10089","reported_seq":10275,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.937728+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:37.937728+0000","last_peered":"2024-09-18T11:58:37.937728+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:37.937728+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38780952,"num_objects":9469,"num_object_clones":0,"num_object_copies":18938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9469,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6947,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3227},{"shards":"2","objects":5933}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9952","reported_seq":9981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.973480+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:37.973480+0000","last_peered":"2024-09-18T11:58:37.973480+0000","last_clean":"2024-09-18T11:58:37.973480+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:37.973480+0000","last_undegraded":"2024-09-18T11:58:37.973480+0000","last_fullsized":"2024-09-18T11:58:37.973480+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9952,"log_dups_size":0,"ondisk_log_size":9952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38125568,"num_objects":9308,"num_object_clones":0,"num_object_copies":18616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9952,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10025","reported_seq":10216,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.558516+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:37.558516+0000","last_peered":"2024-09-18T11:58:37.558516+0000","last_clean":"2024-09-18T11:58:37.558516+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:37.558516+0000","last_undegraded":"2024-09-18T11:58:37.558516+0000","last_fullsized":"2024-09-18T11:58:37.558516+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38592512,"num_objects":9422,"num_object_clones":0,"num_object_copies":18844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7190,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10035","reported_seq":10116,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.551934+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:37.551934+0000","last_peered":"2024-09-18T11:58:37.551934+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:37.551934+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38454482,"num_objects":9390,"num_object_clones":0,"num_object_copies":18780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10033,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5863},{"shards":"0,2","objects":3206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9949","reported_seq":10036,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.379815+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:37.379815+0000","last_peered":"2024-09-18T11:58:37.379815+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:37.379815+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38597842,"num_objects":9425,"num_object_clones":0,"num_object_copies":18850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9947,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5866},{"shards":"0,2","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10097","reported_seq":10321,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.517675+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:37.517675+0000","last_peered":"2024-09-18T11:58:37.517675+0000","last_clean":"2024-09-18T11:58:37.517675+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:37.517675+0000","last_undegraded":"2024-09-18T11:58:37.517675+0000","last_fullsized":"2024-09-18T11:58:37.517675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38932480,"num_objects":9505,"num_object_clones":0,"num_object_copies":19010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6414,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10034","reported_seq":17161,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.826742+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:37.826742+0000","last_peered":"2024-09-18T11:58:37.826742+0000","last_clean":"2024-09-18T11:58:37.826742+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:37.826742+0000","last_undegraded":"2024-09-18T11:58:37.826742+0000","last_fullsized":"2024-09-18T11:58:37.826742+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38384850,"num_objects":9373,"num_object_clones":0,"num_object_copies":18746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10032,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10017","reported_seq":10095,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.428204+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:37.428204+0000","last_peered":"2024-09-18T11:58:37.428204+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:37.428204+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38622418,"num_objects":9431,"num_object_clones":0,"num_object_copies":18862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10015,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5899},{"shards":"0,2","objects":3242}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10044","reported_seq":10100,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.946306+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:37.946306+0000","last_peered":"2024-09-18T11:58:37.946306+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:37.946306+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38581458,"num_objects":9421,"num_object_clones":0,"num_object_copies":18842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10042,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3240},{"shards":"2","objects":5871}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10073","reported_seq":10282,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.944223+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:37.944223+0000","last_peered":"2024-09-18T11:58:37.944223+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:37.944223+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38604800,"num_objects":9425,"num_object_clones":0,"num_object_copies":18850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3271},{"shards":"2","objects":5832}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10075","reported_seq":10157,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.483720+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:37.483720+0000","last_peered":"2024-09-18T11:58:37.483720+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:37.483720+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38720105,"num_objects":9454,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5919},{"shards":"0,2","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10050","reported_seq":14698,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:37.544497+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:37.544497+0000","last_peered":"2024-09-18T11:58:37.544497+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:37.544497+0000","last_undegraded":"2024-09-18T11:58:37.544497+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38799163,"num_objects":9475,"num_object_clones":0,"num_object_copies":18950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1243,"num_objects_unfound":0,"num_objects_dirty":9475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10047,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2285,"num_bytes_recovered":9334784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8232},{"shards":"0,2","objects":1243}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9908","reported_seq":9989,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.390088+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:37.390088+0000","last_peered":"2024-09-18T11:58:37.390088+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:37.390088+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9908,"log_dups_size":0,"ondisk_log_size":9908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38053074,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9906,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5699},{"shards":"0,2","objects":3286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10053","reported_seq":10138,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.494647+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:37.494647+0000","last_peered":"2024-09-18T11:58:37.494647+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:37.494647+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38746532,"num_objects":9463,"num_object_clones":0,"num_object_copies":18926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5940},{"shards":"0,2","objects":3231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617443404,"num_objects":150761,"num_object_clones":0,"num_object_copies":301522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1243,"num_objects_unfound":0,"num_objects_dirty":150761,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144454,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5785,"num_bytes_recovered":23593577,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1270697984,"data_stored":1270620260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160461,"ondisk_log_size":160461,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051641,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721628,"kb_used_data":415436,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93650212,"statfs":{"total":96636764160,"available":95897817088,"internally_reserved":0,"allocated":425406464,"data_stored":425178817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.41799999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":11.109}]}]},{"osd":1,"up_from":35,"seq":150323855404,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619648,"kb_used_data":317624,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93752192,"statfs":{"total":96636764160,"available":96002244608,"internally_reserved":0,"allocated":325246976,"data_stored":324956564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.193},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.27100000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.22500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247868,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":807380,"kb_used_data":509576,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93564460,"statfs":{"total":96636764160,"available":95810007040,"internally_reserved":0,"allocated":521805824,"data_stored":521578627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,3,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.23899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":521129984,"data_stored":521126505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324820992,"data_stored":324754298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":424747008,"data_stored":424739457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:40.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:58:40.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'10038", 'reported_seq': 10109, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:37.489925+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:58:37.489925+0000', 'last_peered': '2024-09-18T11:58:37.489925+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:58:37.489925+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38903808, 'num_objects': 9498, 'num_object_clones': 0, 'num_object_copies': 18996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10038, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5899}, {'shards': '0,2', 'objects': 3329}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:40.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:58:40.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'10089", 'reported_seq': 10275, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:37.937728+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:58:37.937728+0000', 'last_peered': '2024-09-18T11:58:37.937728+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:58:37.937728+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38780952, 'num_objects': 9469, 'num_object_clones': 0, 'num_object_copies': 18938, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9469, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6947, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3227}, {'shards': '2', 'objects': 5933}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:40.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:58:40.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'10035", 'reported_seq': 10116, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:37.551934+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:58:37.551934+0000', 'last_peered': '2024-09-18T11:58:37.551934+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:58:37.551934+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38454482, 'num_objects': 9390, 'num_object_clones': 0, 'num_object_copies': 18780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10033, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5863}, {'shards': '0,2', 'objects': 3206}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:40.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:58:40.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'9949", 'reported_seq': 10036, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:37.379815+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:58:37.379815+0000', 'last_peered': '2024-09-18T11:58:37.379815+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:58:37.379815+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9949, 'log_dups_size': 0, 'ondisk_log_size': 9949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38597842, 'num_objects': 9425, 'num_object_clones': 0, 'num_object_copies': 18850, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9947, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5866}, {'shards': '0,2', 'objects': 3299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:40.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:58:40.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'10017", 'reported_seq': 10095, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:37.428204+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:58:37.428204+0000', 'last_peered': '2024-09-18T11:58:37.428204+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:58:37.428204+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38622418, 'num_objects': 9431, 'num_object_clones': 0, 'num_object_copies': 18862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10015, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5899}, {'shards': '0,2', 'objects': 3242}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:40.422 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:58:40.422 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'10044", 'reported_seq': 10100, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:37.946306+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:58:37.946306+0000', 'last_peered': '2024-09-18T11:58:37.946306+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:58:37.946306+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38581458, 'num_objects': 9421, 'num_object_clones': 0, 'num_object_copies': 18842, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10042, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3240}, {'shards': '2', 'objects': 5871}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:40.422 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:58:40.422 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'10073", 'reported_seq': 10282, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:37.944223+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:58:37.944223+0000', 'last_peered': '2024-09-18T11:58:37.944223+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:58:37.944223+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38604800, 'num_objects': 9425, 'num_object_clones': 0, 'num_object_copies': 18850, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6553, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3271}, {'shards': '2', 'objects': 5832}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:40.422 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:58:40.422 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'10075", 'reported_seq': 10157, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:37.483720+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:58:37.483720+0000', 'last_peered': '2024-09-18T11:58:37.483720+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:58:37.483720+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38720105, 'num_objects': 9454, 'num_object_clones': 0, 'num_object_copies': 18908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9454, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10074, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5919}, {'shards': '0,2', 'objects': 3225}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:40.422 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:58:40.422 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'10050", 'reported_seq': 14698, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:58:37.544497+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:58:37.544497+0000', 'last_peered': '2024-09-18T11:58:37.544497+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:58:37.544497+0000', 'last_undegraded': '2024-09-18T11:58:37.544497+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38799163, 'num_objects': 9475, 'num_object_clones': 0, 'num_object_copies': 18950, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1243, 'num_objects_unfound': 0, 'num_objects_dirty': 9475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10047, 'num_write_kb': 39032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2285, 'num_bytes_recovered': 9334784, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 8232}, {'shards': '0,2', 'objects': 1243}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:40.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:58:40.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'9908", 'reported_seq': 9989, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:37.390088+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:58:37.390088+0000', 'last_peered': '2024-09-18T11:58:37.390088+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:58:37.390088+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 9908, 'log_dups_size': 0, 'ondisk_log_size': 9908, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38053074, 'num_objects': 9292, 'num_object_clones': 0, 'num_object_copies': 18584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9906, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5699}, {'shards': '0,2', 'objects': 3286}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:40.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:58:40.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'10053", 'reported_seq': 10138, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:37.494647+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:58:37.494647+0000', 'last_peered': '2024-09-18T11:58:37.494647+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:58:37.494647+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38746532, 'num_objects': 9463, 'num_object_clones': 0, 'num_object_copies': 18926, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9463, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10050, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5940}, {'shards': '0,2', 'objects': 3231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:40.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:41.428 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:41.428 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/301526 objects degraded (11.795%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":321,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":150763,"data_bytes":617443423,"bytes_used":2200223744,"bytes_avail":287710068736,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":301526,"degraded_ratio":0.11794671106305923,"misplaced_objects":1243,"misplaced_total":301526,"misplaced_ratio":0.0041223642405630026,"recovering_objects_per_sec":34,"recovering_bytes_per_sec":141253,"recovering_keys_per_sec":0,"num_objects_recovered":417,"num_bytes_recovered":1695744,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":495494,"read_op_per_sec":0,"write_op_per_sec":522},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:41.472 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:41.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:42.099 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:42.107 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:42.126 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-09-18T11:58:41.115464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617443423,"num_objects":150763,"num_object_clones":0,"num_object_copies":301526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1243,"num_objects_unfound":0,"num_objects_dirty":150763,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144456,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5786,"num_bytes_recovered":23593596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160464,"ondisk_log_size":160464,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2148656,"kb_used_data":1242636,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":280966864,"statfs":{"total":289910292480,"available":287710068736,"internally_reserved":0,"allocated":1272459264,"data_stored":1271714008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,5,5,11,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17285120,"num_objects":-4220,"num_object_clones":0,"num_object_copies":-8440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-294,"num_objects_unfound":0,"num_objects_dirty":-4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4220,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004454"},"pg_stats":[{"pgid":"2.f","version":"39'10038","reported_seq":10109,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.489925+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:37.489925+0000","last_peered":"2024-09-18T11:58:37.489925+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:37.489925+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38903808,"num_objects":9498,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10038,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5899},{"shards":"0,2","objects":3329}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10022","reported_seq":10178,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.960060+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:37.960060+0000","last_peered":"2024-09-18T11:58:37.960060+0000","last_clean":"2024-09-18T11:58:37.960060+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:37.960060+0000","last_undegraded":"2024-09-18T11:58:37.960060+0000","last_fullsized":"2024-09-18T11:58:37.960060+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38543360,"num_objects":9410,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7214,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10089","reported_seq":10275,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.937728+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:37.937728+0000","last_peered":"2024-09-18T11:58:37.937728+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:37.937728+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38780952,"num_objects":9469,"num_object_clones":0,"num_object_copies":18938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9469,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6947,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3227},{"shards":"2","objects":5933}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'9952","reported_seq":9981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.973480+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:37.973480+0000","last_peered":"2024-09-18T11:58:37.973480+0000","last_clean":"2024-09-18T11:58:37.973480+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:37.973480+0000","last_undegraded":"2024-09-18T11:58:37.973480+0000","last_fullsized":"2024-09-18T11:58:37.973480+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9952,"log_dups_size":0,"ondisk_log_size":9952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38125568,"num_objects":9308,"num_object_clones":0,"num_object_copies":18616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9952,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10025","reported_seq":10216,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.558516+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:37.558516+0000","last_peered":"2024-09-18T11:58:37.558516+0000","last_clean":"2024-09-18T11:58:37.558516+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:37.558516+0000","last_undegraded":"2024-09-18T11:58:37.558516+0000","last_fullsized":"2024-09-18T11:58:37.558516+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38592512,"num_objects":9422,"num_object_clones":0,"num_object_copies":18844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7190,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10035","reported_seq":10116,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.551934+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:37.551934+0000","last_peered":"2024-09-18T11:58:37.551934+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:37.551934+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38454482,"num_objects":9390,"num_object_clones":0,"num_object_copies":18780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10033,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5863},{"shards":"0,2","objects":3206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'9949","reported_seq":10036,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.379815+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:37.379815+0000","last_peered":"2024-09-18T11:58:37.379815+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:37.379815+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9949,"log_dups_size":0,"ondisk_log_size":9949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38597842,"num_objects":9425,"num_object_clones":0,"num_object_copies":18850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9947,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5866},{"shards":"0,2","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10097","reported_seq":10321,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.517675+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:37.517675+0000","last_peered":"2024-09-18T11:58:37.517675+0000","last_clean":"2024-09-18T11:58:37.517675+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:37.517675+0000","last_undegraded":"2024-09-18T11:58:37.517675+0000","last_fullsized":"2024-09-18T11:58:37.517675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38932480,"num_objects":9505,"num_object_clones":0,"num_object_copies":19010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6414,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10034","reported_seq":17161,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:37.826742+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:37.826742+0000","last_peered":"2024-09-18T11:58:37.826742+0000","last_clean":"2024-09-18T11:58:37.826742+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:37.826742+0000","last_undegraded":"2024-09-18T11:58:37.826742+0000","last_fullsized":"2024-09-18T11:58:37.826742+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38384850,"num_objects":9373,"num_object_clones":0,"num_object_copies":18746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10032,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10017","reported_seq":10095,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.428204+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:37.428204+0000","last_peered":"2024-09-18T11:58:37.428204+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:37.428204+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38622418,"num_objects":9431,"num_object_clones":0,"num_object_copies":18862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10015,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5899},{"shards":"0,2","objects":3242}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10044","reported_seq":10100,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.946306+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:37.946306+0000","last_peered":"2024-09-18T11:58:37.946306+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:37.946306+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38581458,"num_objects":9421,"num_object_clones":0,"num_object_copies":18842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10042,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3240},{"shards":"2","objects":5871}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10073","reported_seq":10282,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.944223+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:37.944223+0000","last_peered":"2024-09-18T11:58:37.944223+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:37.944223+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38604800,"num_objects":9425,"num_object_clones":0,"num_object_copies":18850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3271},{"shards":"2","objects":5832}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10075","reported_seq":10157,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.483720+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:37.483720+0000","last_peered":"2024-09-18T11:58:37.483720+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:37.483720+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38720105,"num_objects":9454,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5919},{"shards":"0,2","objects":3225}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10050","reported_seq":14698,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:37.544497+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:37.544497+0000","last_peered":"2024-09-18T11:58:37.544497+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:37.544497+0000","last_undegraded":"2024-09-18T11:58:37.544497+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38799163,"num_objects":9475,"num_object_clones":0,"num_object_copies":18950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1243,"num_objects_unfound":0,"num_objects_dirty":9475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10047,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2285,"num_bytes_recovered":9334784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8232},{"shards":"0,2","objects":1243}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'9908","reported_seq":9989,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.390088+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:37.390088+0000","last_peered":"2024-09-18T11:58:37.390088+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:37.390088+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":9908,"log_dups_size":0,"ondisk_log_size":9908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38053074,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9906,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5699},{"shards":"0,2","objects":3286}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10053","reported_seq":10138,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:37.494647+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:37.494647+0000","last_peered":"2024-09-18T11:58:37.494647+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:37.494647+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38746532,"num_objects":9463,"num_object_clones":0,"num_object_copies":18926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5940},{"shards":"0,2","objects":3231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617443404,"num_objects":150761,"num_object_clones":0,"num_object_copies":301522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1243,"num_objects_unfound":0,"num_objects_dirty":150761,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144454,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5785,"num_bytes_recovered":23593577,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1270697984,"data_stored":1270620260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160461,"ondisk_log_size":160461,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051641,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721628,"kb_used_data":415436,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93650212,"statfs":{"total":96636764160,"available":95897817088,"internally_reserved":0,"allocated":425406464,"data_stored":425178817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.41799999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":11.109}]}]},{"osd":1,"up_from":35,"seq":150323855404,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619648,"kb_used_data":317624,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93752192,"statfs":{"total":96636764160,"available":96002244608,"internally_reserved":0,"allocated":325246976,"data_stored":324956564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.193},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.27100000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.22500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247868,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":807380,"kb_used_data":509576,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93564460,"statfs":{"total":96636764160,"available":95810007040,"internally_reserved":0,"allocated":521805824,"data_stored":521578627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,3,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.23899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":521129984,"data_stored":521126505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324820992,"data_stored":324754298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":424747008,"data_stored":424739457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:45.129 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:45.906 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:45.915 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-18T11:58:45.120010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":608968799,"num_objects":148694,"num_object_clones":0,"num_object_copies":297388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":148694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146525,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5917,"num_bytes_recovered":24130172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2154300,"kb_used_data":1231652,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":280961220,"statfs":{"total":289910292480,"available":287704289280,"internally_reserved":0,"allocated":1261211648,"data_stored":1260466392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,3,8,9,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16994304,"num_objects":-4149,"num_object_clones":0,"num_object_copies":-8298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-278,"num_objects_unfound":0,"num_objects_dirty":-4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4149,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008068"},"pg_stats":[{"pgid":"2.f","version":"39'10164","reported_seq":10235,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.484358+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:42.484358+0000","last_peered":"2024-09-18T11:58:42.484358+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:42.484358+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38387712,"num_objects":9372,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10164,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5773},{"shards":"0,2","objects":3203}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10164","reported_seq":10320,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:43.010154+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:43.010154+0000","last_peered":"2024-09-18T11:58:43.010154+0000","last_clean":"2024-09-18T11:58:43.010154+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:43.010154+0000","last_undegraded":"2024-09-18T11:58:43.010154+0000","last_fullsized":"2024-09-18T11:58:43.010154+0000","mapping_epoch":29,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37961728,"num_objects":9268,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7356,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10229","reported_seq":10415,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.937203+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:42.937203+0000","last_peered":"2024-09-18T11:58:42.937203+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:42.937203+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38207512,"num_objects":9329,"num_object_clones":0,"num_object_copies":18658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7087,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3087},{"shards":"2","objects":5793}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10074","reported_seq":10103,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:42.887677+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:42.887677+0000","last_peered":"2024-09-18T11:58:42.887677+0000","last_clean":"2024-09-18T11:58:42.887677+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:42.887677+0000","last_undegraded":"2024-09-18T11:58:42.887677+0000","last_fullsized":"2024-09-18T11:58:42.887677+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37625856,"num_objects":9186,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10155","reported_seq":10346,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:42.632398+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:42.632398+0000","last_peered":"2024-09-18T11:58:42.632398+0000","last_clean":"2024-09-18T11:58:42.632398+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:42.632398+0000","last_undegraded":"2024-09-18T11:58:42.632398+0000","last_fullsized":"2024-09-18T11:58:42.632398+0000","mapping_epoch":29,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38060032,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7320,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10161","reported_seq":10242,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.615546+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:42.615546+0000","last_peered":"2024-09-18T11:58:42.615546+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:42.615546+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37938386,"num_objects":9264,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5737},{"shards":"0,2","objects":3080}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10082","reported_seq":10169,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.562620+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:42.562620+0000","last_peered":"2024-09-18T11:58:42.562620+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:42.562620+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38053074,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10080,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5733},{"shards":"0,2","objects":3166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10221","reported_seq":10446,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:42.586612+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:42.586612+0000","last_peered":"2024-09-18T11:58:42.586612+0000","last_clean":"2024-09-18T11:58:42.586612+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:42.586612+0000","last_undegraded":"2024-09-18T11:58:42.586612+0000","last_fullsized":"2024-09-18T11:58:42.586612+0000","mapping_epoch":29,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38424576,"num_objects":9381,"num_object_clones":0,"num_object_copies":18762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10150","reported_seq":17278,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:42.877375+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:42.877375+0000","last_peered":"2024-09-18T11:58:42.877375+0000","last_clean":"2024-09-18T11:58:42.877375+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:42.877375+0000","last_undegraded":"2024-09-18T11:58:42.877375+0000","last_fullsized":"2024-09-18T11:58:42.877375+0000","mapping_epoch":38,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37909714,"num_objects":9257,"num_object_clones":0,"num_object_copies":18514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10148,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10159","reported_seq":10236,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.632683+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:42.632683+0000","last_peered":"2024-09-18T11:58:42.632683+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:42.632683+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38040786,"num_objects":9289,"num_object_clones":0,"num_object_copies":18578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10157,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5757},{"shards":"0,2","objects":3100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10170","reported_seq":10226,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.986041+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:42.986041+0000","last_peered":"2024-09-18T11:58:42.986041+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:42.986041+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38065362,"num_objects":9295,"num_object_clones":0,"num_object_copies":18590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10168,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3114},{"shards":"2","objects":5745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10202","reported_seq":10411,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:43.011627+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:43.011627+0000","last_peered":"2024-09-18T11:58:43.011627+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:43.011627+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38076416,"num_objects":9296,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3142},{"shards":"2","objects":5703}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10199","reported_seq":10279,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.412237+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:42.412237+0000","last_peered":"2024-09-18T11:58:42.412237+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:42.412237+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38212201,"num_objects":9330,"num_object_clones":0,"num_object_copies":18660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5795},{"shards":"0,2","objects":3101}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10183","reported_seq":15094,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:42.620977+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:42.620977+0000","last_peered":"2024-09-18T11:58:42.620977+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:42.620977+0000","last_undegraded":"2024-09-18T11:58:42.620977+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38254395,"num_objects":9342,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":9342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10180,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2416,"num_bytes_recovered":9871360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8230},{"shards":"0,2","objects":1112}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10032","reported_seq":10113,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.578794+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:42.578794+0000","last_peered":"2024-09-18T11:58:42.578794+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:42.578794+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37545170,"num_objects":9168,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10030,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5575},{"shards":"0,2","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10185","reported_seq":10270,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.586731+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:42.586731+0000","last_peered":"2024-09-18T11:58:42.586731+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:42.586731+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38205860,"num_objects":9331,"num_object_clones":0,"num_object_copies":18662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10182,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5808},{"shards":"0,2","objects":3099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":608968780,"num_objects":148692,"num_object_clones":0,"num_object_copies":297384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":148692,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146523,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5916,"num_bytes_recovered":24130153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1259626496,"data_stored":1259548772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160930,"ondisk_log_size":160930,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051642,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721616,"kb_used_data":411272,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93650224,"statfs":{"total":96636764160,"available":95897829376,"internally_reserved":0,"allocated":421142528,"data_stored":420914881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.249}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.23200000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855405,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623280,"kb_used_data":317096,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93748560,"statfs":{"total":96636764160,"available":95998525440,"internally_reserved":0,"allocated":324706304,"data_stored":324415892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.25800000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":11.147},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.29099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":809404,"kb_used_data":503284,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93562436,"statfs":{"total":96636764160,"available":95807934464,"internally_reserved":0,"allocated":515362816,"data_stored":515135619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,4,5,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.40100000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":4.4740000000000002},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.39100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":514732032,"data_stored":514728553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324370432,"data_stored":324303738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":420524032,"data_stored":420516481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:45.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:46.731 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:46.738 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:46.745 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-18T11:58:45.120010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":608968799,"num_objects":148694,"num_object_clones":0,"num_object_copies":297388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":148694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146525,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5917,"num_bytes_recovered":24130172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2154300,"kb_used_data":1231652,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":280961220,"statfs":{"total":289910292480,"available":287704289280,"internally_reserved":0,"allocated":1261211648,"data_stored":1260466392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,3,8,9,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16994304,"num_objects":-4149,"num_object_clones":0,"num_object_copies":-8298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-278,"num_objects_unfound":0,"num_objects_dirty":-4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4149,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008068"},"pg_stats":[{"pgid":"2.f","version":"39'10164","reported_seq":10235,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.484358+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:42.484358+0000","last_peered":"2024-09-18T11:58:42.484358+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:42.484358+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38387712,"num_objects":9372,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10164,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5773},{"shards":"0,2","objects":3203}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10164","reported_seq":10320,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:43.010154+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:43.010154+0000","last_peered":"2024-09-18T11:58:43.010154+0000","last_clean":"2024-09-18T11:58:43.010154+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:43.010154+0000","last_undegraded":"2024-09-18T11:58:43.010154+0000","last_fullsized":"2024-09-18T11:58:43.010154+0000","mapping_epoch":29,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37961728,"num_objects":9268,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7356,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10229","reported_seq":10415,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.937203+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:42.937203+0000","last_peered":"2024-09-18T11:58:42.937203+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:42.937203+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38207512,"num_objects":9329,"num_object_clones":0,"num_object_copies":18658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7087,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3087},{"shards":"2","objects":5793}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10074","reported_seq":10103,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:42.887677+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:42.887677+0000","last_peered":"2024-09-18T11:58:42.887677+0000","last_clean":"2024-09-18T11:58:42.887677+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:42.887677+0000","last_undegraded":"2024-09-18T11:58:42.887677+0000","last_fullsized":"2024-09-18T11:58:42.887677+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37625856,"num_objects":9186,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10155","reported_seq":10346,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:42.632398+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:42.632398+0000","last_peered":"2024-09-18T11:58:42.632398+0000","last_clean":"2024-09-18T11:58:42.632398+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:42.632398+0000","last_undegraded":"2024-09-18T11:58:42.632398+0000","last_fullsized":"2024-09-18T11:58:42.632398+0000","mapping_epoch":29,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38060032,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7320,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10161","reported_seq":10242,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.615546+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:42.615546+0000","last_peered":"2024-09-18T11:58:42.615546+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:42.615546+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37938386,"num_objects":9264,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5737},{"shards":"0,2","objects":3080}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10082","reported_seq":10169,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.562620+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:42.562620+0000","last_peered":"2024-09-18T11:58:42.562620+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:42.562620+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38053074,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10080,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5733},{"shards":"0,2","objects":3166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10221","reported_seq":10446,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:42.586612+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:42.586612+0000","last_peered":"2024-09-18T11:58:42.586612+0000","last_clean":"2024-09-18T11:58:42.586612+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:42.586612+0000","last_undegraded":"2024-09-18T11:58:42.586612+0000","last_fullsized":"2024-09-18T11:58:42.586612+0000","mapping_epoch":29,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38424576,"num_objects":9381,"num_object_clones":0,"num_object_copies":18762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10150","reported_seq":17278,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:42.877375+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:42.877375+0000","last_peered":"2024-09-18T11:58:42.877375+0000","last_clean":"2024-09-18T11:58:42.877375+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:42.877375+0000","last_undegraded":"2024-09-18T11:58:42.877375+0000","last_fullsized":"2024-09-18T11:58:42.877375+0000","mapping_epoch":38,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37909714,"num_objects":9257,"num_object_clones":0,"num_object_copies":18514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10148,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10159","reported_seq":10236,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.632683+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:42.632683+0000","last_peered":"2024-09-18T11:58:42.632683+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:42.632683+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38040786,"num_objects":9289,"num_object_clones":0,"num_object_copies":18578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10157,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5757},{"shards":"0,2","objects":3100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10170","reported_seq":10226,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.986041+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:42.986041+0000","last_peered":"2024-09-18T11:58:42.986041+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:42.986041+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38065362,"num_objects":9295,"num_object_clones":0,"num_object_copies":18590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10168,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3114},{"shards":"2","objects":5745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10202","reported_seq":10411,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:43.011627+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:43.011627+0000","last_peered":"2024-09-18T11:58:43.011627+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:43.011627+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38076416,"num_objects":9296,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3142},{"shards":"2","objects":5703}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10199","reported_seq":10279,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.412237+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:42.412237+0000","last_peered":"2024-09-18T11:58:42.412237+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:42.412237+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38212201,"num_objects":9330,"num_object_clones":0,"num_object_copies":18660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5795},{"shards":"0,2","objects":3101}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10183","reported_seq":15094,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:42.620977+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:42.620977+0000","last_peered":"2024-09-18T11:58:42.620977+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:42.620977+0000","last_undegraded":"2024-09-18T11:58:42.620977+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38254395,"num_objects":9342,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":9342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10180,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2416,"num_bytes_recovered":9871360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8230},{"shards":"0,2","objects":1112}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10032","reported_seq":10113,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.578794+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:42.578794+0000","last_peered":"2024-09-18T11:58:42.578794+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:42.578794+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37545170,"num_objects":9168,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10030,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5575},{"shards":"0,2","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10185","reported_seq":10270,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.586731+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:42.586731+0000","last_peered":"2024-09-18T11:58:42.586731+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:42.586731+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38205860,"num_objects":9331,"num_object_clones":0,"num_object_copies":18662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10182,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5808},{"shards":"0,2","objects":3099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":608968780,"num_objects":148692,"num_object_clones":0,"num_object_copies":297384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":148692,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146523,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5916,"num_bytes_recovered":24130153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1259626496,"data_stored":1259548772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160930,"ondisk_log_size":160930,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051642,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721616,"kb_used_data":411272,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93650224,"statfs":{"total":96636764160,"available":95897829376,"internally_reserved":0,"allocated":421142528,"data_stored":420914881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.249}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.23200000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855405,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623280,"kb_used_data":317096,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93748560,"statfs":{"total":96636764160,"available":95998525440,"internally_reserved":0,"allocated":324706304,"data_stored":324415892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.25800000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":11.147},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.29099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":809404,"kb_used_data":503284,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93562436,"statfs":{"total":96636764160,"available":95807934464,"internally_reserved":0,"allocated":515362816,"data_stored":515135619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,4,5,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.40100000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":4.4740000000000002},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.39100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":514732032,"data_stored":514728553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324370432,"data_stored":324303738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":420524032,"data_stored":420516481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:58:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'10164", 'reported_seq': 10235, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:42.484358+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:58:42.484358+0000', 'last_peered': '2024-09-18T11:58:42.484358+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:58:42.484358+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'100", 'ondisk_log_start': "30'100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38387712, 'num_objects': 9372, 'num_object_clones': 0, 'num_object_copies': 18744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10164, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5773}, {'shards': '0,2', 'objects': 3203}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:58:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'10229", 'reported_seq': 10415, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:42.937203+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:58:42.937203+0000', 'last_peered': '2024-09-18T11:58:42.937203+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:58:42.937203+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'200", 'ondisk_log_start': "30'200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38207512, 'num_objects': 9329, 'num_object_clones': 0, 'num_object_copies': 18658, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9329, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7087, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3087}, {'shards': '2', 'objects': 5793}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:58:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'10161", 'reported_seq': 10242, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:42.615546+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:58:42.615546+0000', 'last_peered': '2024-09-18T11:58:42.615546+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:58:42.615546+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'100", 'ondisk_log_start': "30'100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37938386, 'num_objects': 9264, 'num_object_clones': 0, 'num_object_copies': 18528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10159, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5737}, {'shards': '0,2', 'objects': 3080}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:58:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'10082", 'reported_seq': 10169, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:42.562620+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:58:42.562620+0000', 'last_peered': '2024-09-18T11:58:42.562620+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:58:42.562620+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38053074, 'num_objects': 9292, 'num_object_clones': 0, 'num_object_copies': 18584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10080, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5733}, {'shards': '0,2', 'objects': 3166}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:58:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'10159", 'reported_seq': 10236, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:42.632683+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:58:42.632683+0000', 'last_peered': '2024-09-18T11:58:42.632683+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:58:42.632683+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'100", 'ondisk_log_start': "30'100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38040786, 'num_objects': 9289, 'num_object_clones': 0, 'num_object_copies': 18578, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10157, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5757}, {'shards': '0,2', 'objects': 3100}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:58:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'10170", 'reported_seq': 10226, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:42.986041+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:58:42.986041+0000', 'last_peered': '2024-09-18T11:58:42.986041+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:58:42.986041+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'100", 'ondisk_log_start': "30'100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38065362, 'num_objects': 9295, 'num_object_clones': 0, 'num_object_copies': 18590, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10168, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3114}, {'shards': '2', 'objects': 5745}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:58:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'10202", 'reported_seq': 10411, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:43.011627+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:58:43.011627+0000', 'last_peered': '2024-09-18T11:58:43.011627+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:58:43.011627+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'200", 'ondisk_log_start': "30'200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38076416, 'num_objects': 9296, 'num_object_clones': 0, 'num_object_copies': 18592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9296, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6682, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3142}, {'shards': '2', 'objects': 5703}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:58:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'10199", 'reported_seq': 10279, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:42.412237+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:58:42.412237+0000', 'last_peered': '2024-09-18T11:58:42.412237+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:58:42.412237+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'100", 'ondisk_log_start': "30'100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38212201, 'num_objects': 9330, 'num_object_clones': 0, 'num_object_copies': 18660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10198, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5795}, {'shards': '0,2', 'objects': 3101}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:46.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:58:46.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'10183", 'reported_seq': 15094, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:58:42.620977+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:58:42.620977+0000', 'last_peered': '2024-09-18T11:58:42.620977+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:58:42.620977+0000', 'last_undegraded': '2024-09-18T11:58:42.620977+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "30'100", 'ondisk_log_start': "30'100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38254395, 'num_objects': 9342, 'num_object_clones': 0, 'num_object_copies': 18684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1112, 'num_objects_unfound': 0, 'num_objects_dirty': 9342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10180, 'num_write_kb': 39032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2416, 'num_bytes_recovered': 9871360, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 8230}, {'shards': '0,2', 'objects': 1112}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:46.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:58:46.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'10032", 'reported_seq': 10113, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:42.578794+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:58:42.578794+0000', 'last_peered': '2024-09-18T11:58:42.578794+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:58:42.578794+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37545170, 'num_objects': 9168, 'num_object_clones': 0, 'num_object_copies': 18336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10030, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5575}, {'shards': '0,2', 'objects': 3162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:46.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:58:46.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'10185", 'reported_seq': 10270, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:42.586731+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:58:42.586731+0000', 'last_peered': '2024-09-18T11:58:42.586731+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:58:42.586731+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'100", 'ondisk_log_start': "30'100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38205860, 'num_objects': 9331, 'num_object_clones': 0, 'num_object_copies': 18662, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10182, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5808}, {'shards': '0,2', 'objects': 3099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:46.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:47.652 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/297388 objects degraded (11.959%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":327,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":148694,"data_bytes":608968799,"bytes_used":2206003200,"bytes_avail":287704289280,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":297388,"degraded_ratio":0.11958787846180748,"misplaced_objects":1112,"misplaced_total":297388,"misplaced_ratio":0.0037392228334700794,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":94840,"recovering_keys_per_sec":0,"num_objects_recovered":278,"num_bytes_recovered":1138688,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":345},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:47.706 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:47.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:48.494 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:48.513 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:48.537 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-18T11:58:47.120472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":608968799,"num_objects":148694,"num_object_clones":0,"num_object_copies":297388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":148694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146525,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5917,"num_bytes_recovered":24130172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2154300,"kb_used_data":1231652,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":280961220,"statfs":{"total":289910292480,"available":287704289280,"internally_reserved":0,"allocated":1261211648,"data_stored":1260466392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,3,8,9,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16994304,"num_objects":-4149,"num_object_clones":0,"num_object_copies":-8298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-278,"num_objects_unfound":0,"num_objects_dirty":-4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4149,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006363"},"pg_stats":[{"pgid":"2.f","version":"39'10164","reported_seq":10235,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.484358+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:42.484358+0000","last_peered":"2024-09-18T11:58:42.484358+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:42.484358+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38387712,"num_objects":9372,"num_object_clones":0,"num_object_copies":18744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10164,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5773},{"shards":"0,2","objects":3203}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10164","reported_seq":10320,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:43.010154+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:43.010154+0000","last_peered":"2024-09-18T11:58:43.010154+0000","last_clean":"2024-09-18T11:58:43.010154+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:43.010154+0000","last_undegraded":"2024-09-18T11:58:43.010154+0000","last_fullsized":"2024-09-18T11:58:43.010154+0000","mapping_epoch":29,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37961728,"num_objects":9268,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7356,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10229","reported_seq":10415,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.937203+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:42.937203+0000","last_peered":"2024-09-18T11:58:42.937203+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:42.937203+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38207512,"num_objects":9329,"num_object_clones":0,"num_object_copies":18658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7087,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3087},{"shards":"2","objects":5793}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10074","reported_seq":10103,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:42.887677+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:42.887677+0000","last_peered":"2024-09-18T11:58:42.887677+0000","last_clean":"2024-09-18T11:58:42.887677+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:42.887677+0000","last_undegraded":"2024-09-18T11:58:42.887677+0000","last_fullsized":"2024-09-18T11:58:42.887677+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37625856,"num_objects":9186,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10155","reported_seq":10346,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:42.632398+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:42.632398+0000","last_peered":"2024-09-18T11:58:42.632398+0000","last_clean":"2024-09-18T11:58:42.632398+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:42.632398+0000","last_undegraded":"2024-09-18T11:58:42.632398+0000","last_fullsized":"2024-09-18T11:58:42.632398+0000","mapping_epoch":29,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38060032,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7320,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10161","reported_seq":10242,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.615546+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:42.615546+0000","last_peered":"2024-09-18T11:58:42.615546+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:42.615546+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37938386,"num_objects":9264,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10159,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5737},{"shards":"0,2","objects":3080}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10082","reported_seq":10169,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.562620+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:42.562620+0000","last_peered":"2024-09-18T11:58:42.562620+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:42.562620+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38053074,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10080,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5733},{"shards":"0,2","objects":3166}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10221","reported_seq":10446,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:42.586612+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:42.586612+0000","last_peered":"2024-09-18T11:58:42.586612+0000","last_clean":"2024-09-18T11:58:42.586612+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:42.586612+0000","last_undegraded":"2024-09-18T11:58:42.586612+0000","last_fullsized":"2024-09-18T11:58:42.586612+0000","mapping_epoch":29,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38424576,"num_objects":9381,"num_object_clones":0,"num_object_copies":18762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10150","reported_seq":17278,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:42.877375+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:42.877375+0000","last_peered":"2024-09-18T11:58:42.877375+0000","last_clean":"2024-09-18T11:58:42.877375+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:42.877375+0000","last_undegraded":"2024-09-18T11:58:42.877375+0000","last_fullsized":"2024-09-18T11:58:42.877375+0000","mapping_epoch":38,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37909714,"num_objects":9257,"num_object_clones":0,"num_object_copies":18514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10148,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10159","reported_seq":10236,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.632683+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:42.632683+0000","last_peered":"2024-09-18T11:58:42.632683+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:42.632683+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38040786,"num_objects":9289,"num_object_clones":0,"num_object_copies":18578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10157,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5757},{"shards":"0,2","objects":3100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10170","reported_seq":10226,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.986041+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:42.986041+0000","last_peered":"2024-09-18T11:58:42.986041+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:42.986041+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38065362,"num_objects":9295,"num_object_clones":0,"num_object_copies":18590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10168,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3114},{"shards":"2","objects":5745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10202","reported_seq":10411,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:43.011627+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:43.011627+0000","last_peered":"2024-09-18T11:58:43.011627+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:43.011627+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38076416,"num_objects":9296,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3142},{"shards":"2","objects":5703}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10199","reported_seq":10279,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.412237+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:42.412237+0000","last_peered":"2024-09-18T11:58:42.412237+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:42.412237+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38212201,"num_objects":9330,"num_object_clones":0,"num_object_copies":18660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10198,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5795},{"shards":"0,2","objects":3101}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10183","reported_seq":15094,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:42.620977+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:42.620977+0000","last_peered":"2024-09-18T11:58:42.620977+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:42.620977+0000","last_undegraded":"2024-09-18T11:58:42.620977+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38254395,"num_objects":9342,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":9342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10180,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2416,"num_bytes_recovered":9871360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8230},{"shards":"0,2","objects":1112}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10032","reported_seq":10113,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.578794+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:42.578794+0000","last_peered":"2024-09-18T11:58:42.578794+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:42.578794+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37545170,"num_objects":9168,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10030,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5575},{"shards":"0,2","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10185","reported_seq":10270,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:42.586731+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:42.586731+0000","last_peered":"2024-09-18T11:58:42.586731+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:42.586731+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38205860,"num_objects":9331,"num_object_clones":0,"num_object_copies":18662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10182,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5808},{"shards":"0,2","objects":3099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":608968780,"num_objects":148692,"num_object_clones":0,"num_object_copies":297384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":148692,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146523,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5916,"num_bytes_recovered":24130153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1259626496,"data_stored":1259548772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160930,"ondisk_log_size":160930,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051642,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721616,"kb_used_data":411272,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93650224,"statfs":{"total":96636764160,"available":95897829376,"internally_reserved":0,"allocated":421142528,"data_stored":420914881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.249}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.23200000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855405,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623280,"kb_used_data":317096,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93748560,"statfs":{"total":96636764160,"available":95998525440,"internally_reserved":0,"allocated":324706304,"data_stored":324415892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.25800000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":11.147},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.29099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":809404,"kb_used_data":503284,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93562436,"statfs":{"total":96636764160,"available":95807934464,"internally_reserved":0,"allocated":515362816,"data_stored":515135619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,4,5,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.40100000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":4.4740000000000002},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.39100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":514732032,"data_stored":514728553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324370432,"data_stored":324303738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":420524032,"data_stored":420516481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:51.539 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:52.314 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:52.325 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:52.370 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-18T11:58:51.121362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600461407,"num_objects":146617,"num_object_clones":0,"num_object_copies":293234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":967,"num_objects_unfound":0,"num_objects_dirty":146617,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148602,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6062,"num_bytes_recovered":24724092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2143584,"kb_used_data":1220932,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":280971936,"statfs":{"total":289910292480,"available":287715262464,"internally_reserved":0,"allocated":1250234368,"data_stored":1249489112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,5,9,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16982016,"num_objects":-4146,"num_object_clones":0,"num_object_copies":-8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-276,"num_objects_unfound":0,"num_objects_dirty":-4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4146,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006415"},"pg_stats":[{"pgid":"2.f","version":"39'10279","reported_seq":10350,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:47.553528+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:47.553528+0000","last_peered":"2024-09-18T11:58:47.553528+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:47.553528+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37916672,"num_objects":9257,"num_object_clones":0,"num_object_copies":18514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10279,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5658},{"shards":"0,2","objects":3088}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10298","reported_seq":10455,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:47.993500+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:47.993500+0000","last_peered":"2024-09-18T11:58:47.993500+0000","last_clean":"2024-09-18T11:58:47.993500+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:47.993500+0000","last_undegraded":"2024-09-18T11:58:47.993500+0000","last_fullsized":"2024-09-18T11:58:47.993500+0000","mapping_epoch":29,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37412864,"num_objects":9134,"num_object_clones":0,"num_object_copies":18268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7490,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10359","reported_seq":10545,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:48.013243+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:48.013243+0000","last_peered":"2024-09-18T11:58:48.013243+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:48.013243+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37675032,"num_objects":9199,"num_object_clones":0,"num_object_copies":18398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9199,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7217,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2957},{"shards":"2","objects":5663}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10191","reported_seq":10220,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:48.007317+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:48.007317+0000","last_peered":"2024-09-18T11:58:48.007317+0000","last_clean":"2024-09-18T11:58:48.007317+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:48.007317+0000","last_undegraded":"2024-09-18T11:58:48.007317+0000","last_fullsized":"2024-09-18T11:58:48.007317+0000","mapping_epoch":29,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37146624,"num_objects":9069,"num_object_clones":0,"num_object_copies":18138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10191,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10285","reported_seq":10476,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:47.538823+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:47.538823+0000","last_peered":"2024-09-18T11:58:47.538823+0000","last_clean":"2024-09-18T11:58:47.538823+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:47.538823+0000","last_undegraded":"2024-09-18T11:58:47.538823+0000","last_fullsized":"2024-09-18T11:58:47.538823+0000","mapping_epoch":29,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37527552,"num_objects":9162,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7450,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10278","reported_seq":10359,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:47.563210+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:47.563210+0000","last_peered":"2024-09-18T11:58:47.563210+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:47.563210+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37459154,"num_objects":9147,"num_object_clones":0,"num_object_copies":18294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10276,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5620},{"shards":"0,2","objects":2963}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10211","reported_seq":10298,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:47.482222+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:47.482222+0000","last_peered":"2024-09-18T11:58:47.482222+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:47.482222+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37524690,"num_objects":9163,"num_object_clones":0,"num_object_copies":18326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10209,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5604},{"shards":"0,2","objects":3037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10363","reported_seq":10588,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:47.651773+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:47.651773+0000","last_peered":"2024-09-18T11:58:47.651773+0000","last_clean":"2024-09-18T11:58:47.651773+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:47.651773+0000","last_undegraded":"2024-09-18T11:58:47.651773+0000","last_fullsized":"2024-09-18T11:58:47.651773+0000","mapping_epoch":29,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37842944,"num_objects":9239,"num_object_clones":0,"num_object_copies":18478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6680,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10257","reported_seq":17385,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:47.969560+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:47.969560+0000","last_peered":"2024-09-18T11:58:47.969560+0000","last_clean":"2024-09-18T11:58:47.969560+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:47.969560+0000","last_undegraded":"2024-09-18T11:58:47.969560+0000","last_fullsized":"2024-09-18T11:58:47.969560+0000","mapping_epoch":38,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37471442,"num_objects":9150,"num_object_clones":0,"num_object_copies":18300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10255,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10305","reported_seq":10380,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:47.667950+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:47.667950+0000","last_peered":"2024-09-18T11:58:47.667950+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:47.667950+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37442770,"num_objects":9143,"num_object_clones":0,"num_object_copies":18286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10303,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5611},{"shards":"0,2","objects":2954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10312","reported_seq":10367,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:48.010015+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:48.010015+0000","last_peered":"2024-09-18T11:58:48.010015+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:48.010015+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37483730,"num_objects":9153,"num_object_clones":0,"num_object_copies":18306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10310,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2972},{"shards":"2","objects":5603}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10326","reported_seq":10534,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:47.985373+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:47.985373+0000","last_peered":"2024-09-18T11:58:47.985373+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:47.985373+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37568512,"num_objects":9172,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6806,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3018},{"shards":"2","objects":5579}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10348","reported_seq":10430,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:47.593040+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:47.593040+0000","last_peered":"2024-09-18T11:58:47.593040+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:47.593040+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37601897,"num_objects":9181,"num_object_clones":0,"num_object_copies":18362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10347,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5646},{"shards":"0,2","objects":2952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10320","reported_seq":15521,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:47.656813+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:47.656813+0000","last_peered":"2024-09-18T11:58:47.656813+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:47.656813+0000","last_undegraded":"2024-09-18T11:58:47.656813+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37693243,"num_objects":9205,"num_object_clones":0,"num_object_copies":18410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":967,"num_objects_unfound":0,"num_objects_dirty":9205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10317,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2561,"num_bytes_recovered":10465280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8238},{"shards":"0,2","objects":967}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10164","reported_seq":10245,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:47.615033+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:47.615033+0000","last_peered":"2024-09-18T11:58:47.615033+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:47.615033+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'100","ondisk_log_start":"30'100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37004498,"num_objects":9036,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5443},{"shards":"0,2","objects":3030}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10311","reported_seq":10396,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:47.577089+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:47.577089+0000","last_peered":"2024-09-18T11:58:47.577089+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:47.577089+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37689764,"num_objects":9205,"num_object_clones":0,"num_object_copies":18410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10308,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5682},{"shards":"0,2","objects":2973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":600461388,"num_objects":146615,"num_object_clones":0,"num_object_copies":293230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":967,"num_objects_unfound":0,"num_objects_dirty":146615,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148600,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6061,"num_bytes_recovered":24724073,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1248649216,"data_stored":1248571492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160807,"ondisk_log_size":160807,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051643,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721676,"kb_used_data":407172,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93650164,"statfs":{"total":96636764160,"available":95897767936,"internally_reserved":0,"allocated":416944128,"data_stored":416716481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.36199999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.34799999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855406,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614608,"kb_used_data":316744,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93757232,"statfs":{"total":96636764160,"available":96007405568,"internally_reserved":0,"allocated":324345856,"data_stored":324055444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":12.244999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.28799999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":807300,"kb_used_data":497016,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93564540,"statfs":{"total":96636764160,"available":95810088960,"internally_reserved":0,"allocated":508944384,"data_stored":508717187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.316}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":508293120,"data_stored":508289641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":324005888,"data_stored":323939194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":416350208,"data_stored":416342657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:52.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:53.193 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:53.214 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:53.238 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-18T11:58:53.121790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591982687,"num_objects":144547,"num_object_clones":0,"num_object_copies":289094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":805,"num_objects_unfound":0,"num_objects_dirty":144547,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150672,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6224,"num_bytes_recovered":25387644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160780,"ondisk_log_size":160780,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2112192,"kb_used_data":1210344,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281003328,"statfs":{"total":289910292480,"available":287747407872,"internally_reserved":0,"allocated":1239392256,"data_stored":1238647000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,4,6,8,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25460736,"num_objects":-6216,"num_object_clones":0,"num_object_copies":-12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-438,"num_objects_unfound":0,"num_objects_dirty":-6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":438,"num_bytes_recovered":1794048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006325"},"pg_stats":[{"pgid":"2.f","version":"39'10428","reported_seq":10496,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.614016+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:52.614016+0000","last_peered":"2024-09-18T11:58:52.614016+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:52.614016+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37306368,"num_objects":9108,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10428,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5509},{"shards":"0,2","objects":2939}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10420","reported_seq":10577,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:52.952478+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:52.952478+0000","last_peered":"2024-09-18T11:58:52.952478+0000","last_clean":"2024-09-18T11:58:52.952478+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:52.952478+0000","last_undegraded":"2024-09-18T11:58:52.952478+0000","last_fullsized":"2024-09-18T11:58:52.952478+0000","mapping_epoch":29,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36913152,"num_objects":9012,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7612,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10483","reported_seq":10669,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:53.033473+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:53.033473+0000","last_peered":"2024-09-18T11:58:53.033473+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:53.033473+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37167128,"num_objects":9075,"num_object_clones":0,"num_object_copies":18150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9075,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7341,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2833},{"shards":"2","objects":5539}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10326","reported_seq":10355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:53.009420+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:53.009420+0000","last_peered":"2024-09-18T11:58:53.009420+0000","last_clean":"2024-09-18T11:58:53.009420+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:53.009420+0000","last_undegraded":"2024-09-18T11:58:53.009420+0000","last_fullsized":"2024-09-18T11:58:53.009420+0000","mapping_epoch":29,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36593664,"num_objects":8934,"num_object_clones":0,"num_object_copies":17868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10326,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10390","reported_seq":10581,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:52.617391+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:52.617391+0000","last_peered":"2024-09-18T11:58:52.617391+0000","last_clean":"2024-09-18T11:58:52.617391+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:52.617391+0000","last_undegraded":"2024-09-18T11:58:52.617391+0000","last_fullsized":"2024-09-18T11:58:52.617391+0000","mapping_epoch":29,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37097472,"num_objects":9057,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7555,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10413","reported_seq":10494,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.563206+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:52.563206+0000","last_peered":"2024-09-18T11:58:52.563206+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:52.563206+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36906194,"num_objects":9012,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10411,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5485},{"shards":"0,2","objects":2828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10318","reported_seq":10405,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.619182+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:52.619182+0000","last_peered":"2024-09-18T11:58:52.619182+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:52.619182+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37086418,"num_objects":9056,"num_object_clones":0,"num_object_copies":18112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5497},{"shards":"0,2","objects":2930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10507","reported_seq":10732,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:52.640203+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:52.640203+0000","last_peered":"2024-09-18T11:58:52.640203+0000","last_clean":"2024-09-18T11:58:52.640203+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:52.640203+0000","last_undegraded":"2024-09-18T11:58:52.640203+0000","last_fullsized":"2024-09-18T11:58:52.640203+0000","mapping_epoch":29,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37253120,"num_objects":9095,"num_object_clones":0,"num_object_copies":18190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6824,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10387","reported_seq":17515,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:53.003320+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:53.003320+0000","last_peered":"2024-09-18T11:58:53.003320+0000","last_clean":"2024-09-18T11:58:53.003320+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:53.003320+0000","last_undegraded":"2024-09-18T11:58:53.003320+0000","last_fullsized":"2024-09-18T11:58:53.003320+0000","mapping_epoch":38,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36938962,"num_objects":9020,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10385,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10440","reported_seq":10518,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.653512+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:52.653512+0000","last_peered":"2024-09-18T11:58:52.653512+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:52.653512+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36889810,"num_objects":9008,"num_object_clones":0,"num_object_copies":18016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10438,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5476},{"shards":"0,2","objects":2819}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10443","reported_seq":10499,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.975688+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:52.975688+0000","last_peered":"2024-09-18T11:58:52.975688+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:52.975688+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36947154,"num_objects":9022,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2841},{"shards":"2","objects":5472}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10443","reported_seq":10652,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.993352+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:52.993352+0000","last_peered":"2024-09-18T11:58:52.993352+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:52.993352+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37089280,"num_objects":9055,"num_object_clones":0,"num_object_copies":18110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6923,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2901},{"shards":"2","objects":5462}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10494","reported_seq":10576,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.605915+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:52.605915+0000","last_peered":"2024-09-18T11:58:52.605915+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:52.605915+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37003881,"num_objects":9035,"num_object_clones":0,"num_object_copies":18070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5500},{"shards":"0,2","objects":2806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10440","reported_seq":15965,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:52.605593+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:52.605593+0000","last_peered":"2024-09-18T11:58:52.605593+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:52.605593+0000","last_undegraded":"2024-09-18T11:58:52.605593+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37201723,"num_objects":9085,"num_object_clones":0,"num_object_copies":18170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":805,"num_objects_unfound":0,"num_objects_dirty":9085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10437,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2723,"num_bytes_recovered":11128832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8280},{"shards":"0,2","objects":805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10292","reported_seq":10373,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.575661+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:52.575661+0000","last_peered":"2024-09-18T11:58:52.575661+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:52.575661+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36480210,"num_objects":8908,"num_object_clones":0,"num_object_copies":17816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10290,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5315},{"shards":"0,2","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10453","reported_seq":10538,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.662139+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:52.662139+0000","last_peered":"2024-09-18T11:58:52.662139+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:52.662139+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37108132,"num_objects":9063,"num_object_clones":0,"num_object_copies":18126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10450,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5540},{"shards":"0,2","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":591982668,"num_objects":144545,"num_object_clones":0,"num_object_copies":289090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":805,"num_objects_unfound":0,"num_objects_dirty":144545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150670,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6223,"num_bytes_recovered":25387625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1237614592,"data_stored":1237536868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051644,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705148,"kb_used_data":403124,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93666692,"statfs":{"total":96636764160,"available":95914692608,"internally_reserved":0,"allocated":412798976,"data_stored":412571329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.38},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.29299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.35399999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855407,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618396,"kb_used_data":316372,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93753444,"statfs":{"total":96636764160,"available":96003526656,"internally_reserved":0,"allocated":323964928,"data_stored":323674516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.253},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":8.1899999999999995},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":2.8450000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788648,"kb_used_data":490848,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93583192,"statfs":{"total":96636764160,"available":95829188608,"internally_reserved":0,"allocated":502628352,"data_stored":502401155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.34699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":501854208,"data_stored":501850729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323592192,"data_stored":323525498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412168192,"data_stored":412160641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:53.240 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:58:53.240 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'10428", 'reported_seq': 10496, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:52.614016+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:58:52.614016+0000', 'last_peered': '2024-09-18T11:58:52.614016+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:58:52.614016+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'400", 'ondisk_log_start': "30'400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37306368, 'num_objects': 9108, 'num_object_clones': 0, 'num_object_copies': 18216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10428, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5509}, {'shards': '0,2', 'objects': 2939}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:53.241 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:58:53.241 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'10483", 'reported_seq': 10669, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:53.033473+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:58:53.033473+0000', 'last_peered': '2024-09-18T11:58:53.033473+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:58:53.033473+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'400", 'ondisk_log_start': "30'400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37167128, 'num_objects': 9075, 'num_object_clones': 0, 'num_object_copies': 18150, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9075, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7341, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2833}, {'shards': '2', 'objects': 5539}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:53.241 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:58:53.241 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'10413", 'reported_seq': 10494, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:52.563206+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:58:52.563206+0000', 'last_peered': '2024-09-18T11:58:52.563206+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:58:52.563206+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'400", 'ondisk_log_start': "30'400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36906194, 'num_objects': 9012, 'num_object_clones': 0, 'num_object_copies': 18024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10411, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5485}, {'shards': '0,2', 'objects': 2828}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:53.242 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:58:53.242 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'10318", 'reported_seq': 10405, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:52.619182+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:58:52.619182+0000', 'last_peered': '2024-09-18T11:58:52.619182+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:58:52.619182+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'300", 'ondisk_log_start': "30'300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37086418, 'num_objects': 9056, 'num_object_clones': 0, 'num_object_copies': 18112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10316, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5497}, {'shards': '0,2', 'objects': 2930}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:53.242 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:58:53.242 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'10440", 'reported_seq': 10518, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:52.653512+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:58:52.653512+0000', 'last_peered': '2024-09-18T11:58:52.653512+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:58:52.653512+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'400", 'ondisk_log_start': "30'400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36889810, 'num_objects': 9008, 'num_object_clones': 0, 'num_object_copies': 18016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10438, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5476}, {'shards': '0,2', 'objects': 2819}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:53.242 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:58:53.242 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'10443", 'reported_seq': 10499, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:52.975688+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:58:52.975688+0000', 'last_peered': '2024-09-18T11:58:52.975688+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:58:52.975688+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'400", 'ondisk_log_start': "30'400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36947154, 'num_objects': 9022, 'num_object_clones': 0, 'num_object_copies': 18044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9022, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10441, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2841}, {'shards': '2', 'objects': 5472}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:53.242 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:58:53.242 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'10443", 'reported_seq': 10652, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:52.993352+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:58:52.993352+0000', 'last_peered': '2024-09-18T11:58:52.993352+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:58:52.993352+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'400", 'ondisk_log_start': "30'400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37089280, 'num_objects': 9055, 'num_object_clones': 0, 'num_object_copies': 18110, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6923, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2901}, {'shards': '2', 'objects': 5462}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:53.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:58:53.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'10494", 'reported_seq': 10576, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:52.605915+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:58:52.605915+0000', 'last_peered': '2024-09-18T11:58:52.605915+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:58:52.605915+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'400", 'ondisk_log_start': "30'400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37003881, 'num_objects': 9035, 'num_object_clones': 0, 'num_object_copies': 18070, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10493, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5500}, {'shards': '0,2', 'objects': 2806}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:53.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:58:53.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'10440", 'reported_seq': 15965, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:58:52.605593+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:58:52.605593+0000', 'last_peered': '2024-09-18T11:58:52.605593+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:58:52.605593+0000', 'last_undegraded': '2024-09-18T11:58:52.605593+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "30'400", 'ondisk_log_start': "30'400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37201723, 'num_objects': 9085, 'num_object_clones': 0, 'num_object_copies': 18170, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 805, 'num_objects_unfound': 0, 'num_objects_dirty': 9085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10437, 'num_write_kb': 39032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2723, 'num_bytes_recovered': 11128832, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 8280}, {'shards': '0,2', 'objects': 805}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:53.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:58:53.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'10292", 'reported_seq': 10373, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:52.575661+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:58:52.575661+0000', 'last_peered': '2024-09-18T11:58:52.575661+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:58:52.575661+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'200", 'ondisk_log_start': "30'200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36480210, 'num_objects': 8908, 'num_object_clones': 0, 'num_object_copies': 17816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10290, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5315}, {'shards': '0,2', 'objects': 2902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:53.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:58:53.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'10453", 'reported_seq': 10538, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:52.662139+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:58:52.662139+0000', 'last_peered': '2024-09-18T11:58:52.662139+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:58:52.662139+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'400", 'ondisk_log_start': "30'400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37108132, 'num_objects': 9063, 'num_object_clones': 0, 'num_object_copies': 18126, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9063, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10450, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5540}, {'shards': '0,2', 'objects': 2831}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:53.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:58:54.157 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:54.157 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/289094 objects degraded (12.302%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":334,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":144547,"data_bytes":591982687,"bytes_used":2162884608,"bytes_avail":287747407872,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":289094,"degraded_ratio":0.12301881049070545,"misplaced_objects":805,"misplaced_total":289094,"misplaced_ratio":0.0027845614229281821,"recovering_objects_per_sec":36,"recovering_bytes_per_sec":149425,"recovering_keys_per_sec":0,"num_objects_recovered":438,"num_bytes_recovered":1794048,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":517},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:58:54.201 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:58:54.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:54.806 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:54.815 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:54.830 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-18T11:58:53.121790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591982687,"num_objects":144547,"num_object_clones":0,"num_object_copies":289094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":805,"num_objects_unfound":0,"num_objects_dirty":144547,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150672,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6224,"num_bytes_recovered":25387644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160780,"ondisk_log_size":160780,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2112192,"kb_used_data":1210344,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281003328,"statfs":{"total":289910292480,"available":287747407872,"internally_reserved":0,"allocated":1239392256,"data_stored":1238647000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,4,6,8,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25460736,"num_objects":-6216,"num_object_clones":0,"num_object_copies":-12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-438,"num_objects_unfound":0,"num_objects_dirty":-6216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":438,"num_bytes_recovered":1794048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006325"},"pg_stats":[{"pgid":"2.f","version":"39'10428","reported_seq":10496,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.614016+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:52.614016+0000","last_peered":"2024-09-18T11:58:52.614016+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:52.614016+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37306368,"num_objects":9108,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10428,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5509},{"shards":"0,2","objects":2939}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10420","reported_seq":10577,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:52.952478+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:52.952478+0000","last_peered":"2024-09-18T11:58:52.952478+0000","last_clean":"2024-09-18T11:58:52.952478+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:52.952478+0000","last_undegraded":"2024-09-18T11:58:52.952478+0000","last_fullsized":"2024-09-18T11:58:52.952478+0000","mapping_epoch":29,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36913152,"num_objects":9012,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7612,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10483","reported_seq":10669,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:53.033473+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:53.033473+0000","last_peered":"2024-09-18T11:58:53.033473+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:53.033473+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37167128,"num_objects":9075,"num_object_clones":0,"num_object_copies":18150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9075,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7341,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2833},{"shards":"2","objects":5539}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10326","reported_seq":10355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:53.009420+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:53.009420+0000","last_peered":"2024-09-18T11:58:53.009420+0000","last_clean":"2024-09-18T11:58:53.009420+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:53.009420+0000","last_undegraded":"2024-09-18T11:58:53.009420+0000","last_fullsized":"2024-09-18T11:58:53.009420+0000","mapping_epoch":29,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36593664,"num_objects":8934,"num_object_clones":0,"num_object_copies":17868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10326,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10390","reported_seq":10581,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:52.617391+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:52.617391+0000","last_peered":"2024-09-18T11:58:52.617391+0000","last_clean":"2024-09-18T11:58:52.617391+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:52.617391+0000","last_undegraded":"2024-09-18T11:58:52.617391+0000","last_fullsized":"2024-09-18T11:58:52.617391+0000","mapping_epoch":29,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37097472,"num_objects":9057,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7555,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10413","reported_seq":10494,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.563206+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:52.563206+0000","last_peered":"2024-09-18T11:58:52.563206+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:52.563206+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36906194,"num_objects":9012,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10411,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5485},{"shards":"0,2","objects":2828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10318","reported_seq":10405,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.619182+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:52.619182+0000","last_peered":"2024-09-18T11:58:52.619182+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:52.619182+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37086418,"num_objects":9056,"num_object_clones":0,"num_object_copies":18112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5497},{"shards":"0,2","objects":2930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10507","reported_seq":10732,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:52.640203+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:52.640203+0000","last_peered":"2024-09-18T11:58:52.640203+0000","last_clean":"2024-09-18T11:58:52.640203+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:52.640203+0000","last_undegraded":"2024-09-18T11:58:52.640203+0000","last_fullsized":"2024-09-18T11:58:52.640203+0000","mapping_epoch":29,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37253120,"num_objects":9095,"num_object_clones":0,"num_object_copies":18190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6824,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10387","reported_seq":17515,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:53.003320+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:53.003320+0000","last_peered":"2024-09-18T11:58:53.003320+0000","last_clean":"2024-09-18T11:58:53.003320+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:53.003320+0000","last_undegraded":"2024-09-18T11:58:53.003320+0000","last_fullsized":"2024-09-18T11:58:53.003320+0000","mapping_epoch":38,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36938962,"num_objects":9020,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10385,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10440","reported_seq":10518,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.653512+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:52.653512+0000","last_peered":"2024-09-18T11:58:52.653512+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:52.653512+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36889810,"num_objects":9008,"num_object_clones":0,"num_object_copies":18016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10438,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5476},{"shards":"0,2","objects":2819}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10443","reported_seq":10499,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.975688+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:52.975688+0000","last_peered":"2024-09-18T11:58:52.975688+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:52.975688+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36947154,"num_objects":9022,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2841},{"shards":"2","objects":5472}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10443","reported_seq":10652,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.993352+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:52.993352+0000","last_peered":"2024-09-18T11:58:52.993352+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:52.993352+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37089280,"num_objects":9055,"num_object_clones":0,"num_object_copies":18110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6923,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2901},{"shards":"2","objects":5462}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10494","reported_seq":10576,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.605915+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:52.605915+0000","last_peered":"2024-09-18T11:58:52.605915+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:52.605915+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37003881,"num_objects":9035,"num_object_clones":0,"num_object_copies":18070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5500},{"shards":"0,2","objects":2806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10440","reported_seq":15965,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:52.605593+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:52.605593+0000","last_peered":"2024-09-18T11:58:52.605593+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:52.605593+0000","last_undegraded":"2024-09-18T11:58:52.605593+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37201723,"num_objects":9085,"num_object_clones":0,"num_object_copies":18170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":805,"num_objects_unfound":0,"num_objects_dirty":9085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10437,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2723,"num_bytes_recovered":11128832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8280},{"shards":"0,2","objects":805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10292","reported_seq":10373,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.575661+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:52.575661+0000","last_peered":"2024-09-18T11:58:52.575661+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:52.575661+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36480210,"num_objects":8908,"num_object_clones":0,"num_object_copies":17816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10290,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5315},{"shards":"0,2","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10453","reported_seq":10538,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.662139+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:52.662139+0000","last_peered":"2024-09-18T11:58:52.662139+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:52.662139+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37108132,"num_objects":9063,"num_object_clones":0,"num_object_copies":18126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10450,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5540},{"shards":"0,2","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":591982668,"num_objects":144545,"num_object_clones":0,"num_object_copies":289090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":805,"num_objects_unfound":0,"num_objects_dirty":144545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150670,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6223,"num_bytes_recovered":25387625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1237614592,"data_stored":1237536868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051644,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705148,"kb_used_data":403124,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93666692,"statfs":{"total":96636764160,"available":95914692608,"internally_reserved":0,"allocated":412798976,"data_stored":412571329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.38},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.29299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.35399999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855407,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618396,"kb_used_data":316372,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93753444,"statfs":{"total":96636764160,"available":96003526656,"internally_reserved":0,"allocated":323964928,"data_stored":323674516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.253},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":8.1899999999999995},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":2.8450000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788648,"kb_used_data":490848,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93583192,"statfs":{"total":96636764160,"available":95829188608,"internally_reserved":0,"allocated":502628352,"data_stored":502401155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.34699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":501854208,"data_stored":501850729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323592192,"data_stored":323525498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412168192,"data_stored":412160641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:57.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:58.708 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:58.722 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:58.749 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-18T11:58:57.127183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591982687,"num_objects":144547,"num_object_clones":0,"num_object_copies":289094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":805,"num_objects_unfound":0,"num_objects_dirty":144547,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150672,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6224,"num_bytes_recovered":25387644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160780,"ondisk_log_size":160780,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2112192,"kb_used_data":1210344,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281003328,"statfs":{"total":289910292480,"available":287747407872,"internally_reserved":0,"allocated":1239392256,"data_stored":1238647000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,4,6,8,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16986112,"num_objects":-4147,"num_object_clones":0,"num_object_copies":-8294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-307,"num_objects_unfound":0,"num_objects_dirty":-4147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4147,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":1257472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007172"},"pg_stats":[{"pgid":"2.f","version":"39'10428","reported_seq":10496,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.614016+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:52.614016+0000","last_peered":"2024-09-18T11:58:52.614016+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:52.614016+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37306368,"num_objects":9108,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10428,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5509},{"shards":"0,2","objects":2939}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10420","reported_seq":10577,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:52.952478+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:52.952478+0000","last_peered":"2024-09-18T11:58:52.952478+0000","last_clean":"2024-09-18T11:58:52.952478+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:52.952478+0000","last_undegraded":"2024-09-18T11:58:52.952478+0000","last_fullsized":"2024-09-18T11:58:52.952478+0000","mapping_epoch":29,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36913152,"num_objects":9012,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7612,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10483","reported_seq":10669,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:53.033473+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:53.033473+0000","last_peered":"2024-09-18T11:58:53.033473+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:53.033473+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37167128,"num_objects":9075,"num_object_clones":0,"num_object_copies":18150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9075,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7341,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2833},{"shards":"2","objects":5539}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10326","reported_seq":10355,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:53.009420+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:53.009420+0000","last_peered":"2024-09-18T11:58:53.009420+0000","last_clean":"2024-09-18T11:58:53.009420+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:53.009420+0000","last_undegraded":"2024-09-18T11:58:53.009420+0000","last_fullsized":"2024-09-18T11:58:53.009420+0000","mapping_epoch":29,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36593664,"num_objects":8934,"num_object_clones":0,"num_object_copies":17868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10326,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10390","reported_seq":10581,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:52.617391+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:52.617391+0000","last_peered":"2024-09-18T11:58:52.617391+0000","last_clean":"2024-09-18T11:58:52.617391+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:52.617391+0000","last_undegraded":"2024-09-18T11:58:52.617391+0000","last_fullsized":"2024-09-18T11:58:52.617391+0000","mapping_epoch":29,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37097472,"num_objects":9057,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7555,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10413","reported_seq":10494,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.563206+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:52.563206+0000","last_peered":"2024-09-18T11:58:52.563206+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:52.563206+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36906194,"num_objects":9012,"num_object_clones":0,"num_object_copies":18024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10411,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5485},{"shards":"0,2","objects":2828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10318","reported_seq":10405,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.619182+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:52.619182+0000","last_peered":"2024-09-18T11:58:52.619182+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:52.619182+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37086418,"num_objects":9056,"num_object_clones":0,"num_object_copies":18112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5497},{"shards":"0,2","objects":2930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10507","reported_seq":10732,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:52.640203+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:52.640203+0000","last_peered":"2024-09-18T11:58:52.640203+0000","last_clean":"2024-09-18T11:58:52.640203+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:52.640203+0000","last_undegraded":"2024-09-18T11:58:52.640203+0000","last_fullsized":"2024-09-18T11:58:52.640203+0000","mapping_epoch":29,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37253120,"num_objects":9095,"num_object_clones":0,"num_object_copies":18190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6824,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10387","reported_seq":17515,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:53.003320+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:53.003320+0000","last_peered":"2024-09-18T11:58:53.003320+0000","last_clean":"2024-09-18T11:58:53.003320+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:53.003320+0000","last_undegraded":"2024-09-18T11:58:53.003320+0000","last_fullsized":"2024-09-18T11:58:53.003320+0000","mapping_epoch":38,"log_start":"30'300","ondisk_log_start":"30'300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36938962,"num_objects":9020,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10385,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10440","reported_seq":10518,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.653512+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:52.653512+0000","last_peered":"2024-09-18T11:58:52.653512+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:52.653512+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36889810,"num_objects":9008,"num_object_clones":0,"num_object_copies":18016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10438,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5476},{"shards":"0,2","objects":2819}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10443","reported_seq":10499,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.975688+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:52.975688+0000","last_peered":"2024-09-18T11:58:52.975688+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:52.975688+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36947154,"num_objects":9022,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2841},{"shards":"2","objects":5472}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10443","reported_seq":10652,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.993352+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:52.993352+0000","last_peered":"2024-09-18T11:58:52.993352+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:52.993352+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37089280,"num_objects":9055,"num_object_clones":0,"num_object_copies":18110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6923,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2901},{"shards":"2","objects":5462}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10494","reported_seq":10576,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.605915+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:52.605915+0000","last_peered":"2024-09-18T11:58:52.605915+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:52.605915+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37003881,"num_objects":9035,"num_object_clones":0,"num_object_copies":18070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5500},{"shards":"0,2","objects":2806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10440","reported_seq":15965,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:52.605593+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:52.605593+0000","last_peered":"2024-09-18T11:58:52.605593+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:52.605593+0000","last_undegraded":"2024-09-18T11:58:52.605593+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37201723,"num_objects":9085,"num_object_clones":0,"num_object_copies":18170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":805,"num_objects_unfound":0,"num_objects_dirty":9085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10437,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2723,"num_bytes_recovered":11128832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8280},{"shards":"0,2","objects":805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10292","reported_seq":10373,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.575661+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:52.575661+0000","last_peered":"2024-09-18T11:58:52.575661+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:52.575661+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'200","ondisk_log_start":"30'200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36480210,"num_objects":8908,"num_object_clones":0,"num_object_copies":17816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10290,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5315},{"shards":"0,2","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10453","reported_seq":10538,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:52.662139+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:52.662139+0000","last_peered":"2024-09-18T11:58:52.662139+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:52.662139+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37108132,"num_objects":9063,"num_object_clones":0,"num_object_copies":18126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10450,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5540},{"shards":"0,2","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":591982668,"num_objects":144545,"num_object_clones":0,"num_object_copies":289090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":805,"num_objects_unfound":0,"num_objects_dirty":144545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150670,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6223,"num_bytes_recovered":25387625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1237614592,"data_stored":1237536868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051644,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705148,"kb_used_data":403124,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93666692,"statfs":{"total":96636764160,"available":95914692608,"internally_reserved":0,"allocated":412798976,"data_stored":412571329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.38},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.29299999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.35399999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855407,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":618396,"kb_used_data":316372,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93753444,"statfs":{"total":96636764160,"available":96003526656,"internally_reserved":0,"allocated":323964928,"data_stored":323674516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.253},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":8.1899999999999995},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":2.8450000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788648,"kb_used_data":490848,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93583192,"statfs":{"total":96636764160,"available":95829188608,"internally_reserved":0,"allocated":502628352,"data_stored":502401155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.34699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":501854208,"data_stored":501850729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323592192,"data_stored":323525498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412168192,"data_stored":412160641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:58.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:58:59.445 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:58:59.458 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:58:59.474 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-18T11:58:59.127615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583372895,"num_objects":142445,"num_object_clones":0,"num_object_copies":284890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":659,"num_objects_unfound":0,"num_objects_dirty":142445,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152774,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6370,"num_bytes_recovered":25985660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2117672,"kb_used_data":1199180,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":280997848,"statfs":{"total":289910292480,"available":287741796352,"internally_reserved":0,"allocated":1227960320,"data_stored":1227215064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,5,6,11,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25595904,"num_objects":-6249,"num_object_clones":0,"num_object_copies":-12498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-453,"num_objects_unfound":0,"num_objects_dirty":-6249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6249,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":453,"num_bytes_recovered":1855488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007142"},"pg_stats":[{"pgid":"2.f","version":"39'10548","reported_seq":10619,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.680586+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:57.680586+0000","last_peered":"2024-09-18T11:58:57.680586+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:57.680586+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36814848,"num_objects":8988,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10548,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5389},{"shards":"0,2","objects":2819}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10546","reported_seq":10700,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:58.023039+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:58.023039+0000","last_peered":"2024-09-18T11:58:58.023039+0000","last_clean":"2024-09-18T11:58:58.023039+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:58.023039+0000","last_undegraded":"2024-09-18T11:58:58.023039+0000","last_fullsized":"2024-09-18T11:58:58.023039+0000","mapping_epoch":29,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36397056,"num_objects":8886,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7738,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10595","reported_seq":10778,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.979055+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:57.979055+0000","last_peered":"2024-09-18T11:58:57.979055+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:57.979055+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36708376,"num_objects":8963,"num_object_clones":0,"num_object_copies":17926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8963,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7453,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2721},{"shards":"2","objects":5427}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10449","reported_seq":10478,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:57.791166+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:57.791166+0000","last_peered":"2024-09-18T11:58:57.791166+0000","last_clean":"2024-09-18T11:58:57.791166+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:57.791166+0000","last_undegraded":"2024-09-18T11:58:57.791166+0000","last_fullsized":"2024-09-18T11:58:57.791166+0000","mapping_epoch":29,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36089856,"num_objects":8811,"num_object_clones":0,"num_object_copies":17622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10449,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10523","reported_seq":10714,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:57.648970+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:57.648970+0000","last_peered":"2024-09-18T11:58:57.648970+0000","last_clean":"2024-09-18T11:58:57.648970+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:57.648970+0000","last_undegraded":"2024-09-18T11:58:57.648970+0000","last_fullsized":"2024-09-18T11:58:57.648970+0000","mapping_epoch":29,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36552704,"num_objects":8924,"num_object_clones":0,"num_object_copies":17848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7688,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10554","reported_seq":10630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.649267+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:57.649267+0000","last_peered":"2024-09-18T11:58:57.649267+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:57.649267+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36328658,"num_objects":8871,"num_object_clones":0,"num_object_copies":17742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10552,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5344},{"shards":"0,2","objects":2687}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10468","reported_seq":10554,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.653171+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:57.653171+0000","last_peered":"2024-09-18T11:58:57.653171+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:57.653171+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36472018,"num_objects":8906,"num_object_clones":0,"num_object_copies":17812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10466,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5347},{"shards":"0,2","objects":2780}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10655","reported_seq":10880,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:57.656446+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:57.656446+0000","last_peered":"2024-09-18T11:58:57.656446+0000","last_clean":"2024-09-18T11:58:57.656446+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:57.656446+0000","last_undegraded":"2024-09-18T11:58:57.656446+0000","last_fullsized":"2024-09-18T11:58:57.656446+0000","mapping_epoch":29,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36646912,"num_objects":8947,"num_object_clones":0,"num_object_copies":17894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6972,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10526","reported_seq":17654,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:57.971986+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:57.971986+0000","last_peered":"2024-09-18T11:58:57.971986+0000","last_clean":"2024-09-18T11:58:57.971986+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:57.971986+0000","last_undegraded":"2024-09-18T11:58:57.971986+0000","last_fullsized":"2024-09-18T11:58:57.971986+0000","mapping_epoch":38,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36369618,"num_objects":8881,"num_object_clones":0,"num_object_copies":17762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10524,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10566","reported_seq":10641,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.681814+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:57.681814+0000","last_peered":"2024-09-18T11:58:57.681814+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:57.681814+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36373714,"num_objects":8882,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10564,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5350},{"shards":"0,2","objects":2693}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10573","reported_seq":10629,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:58.004537+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:58.004537+0000","last_peered":"2024-09-18T11:58:58.004537+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:58.004537+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36414674,"num_objects":8892,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10571,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2711},{"shards":"2","objects":5342}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10567","reported_seq":10776,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.994223+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:57.994223+0000","last_peered":"2024-09-18T11:58:57.994223+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:57.994223+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36581376,"num_objects":8931,"num_object_clones":0,"num_object_copies":17862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7047,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2777},{"shards":"2","objects":5338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10614","reported_seq":10696,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.675712+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:57.675712+0000","last_peered":"2024-09-18T11:58:57.675712+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:57.675712+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36512361,"num_objects":8915,"num_object_clones":0,"num_object_copies":17830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5380},{"shards":"0,2","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10582","reported_seq":16399,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:57.686289+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:57.686289+0000","last_peered":"2024-09-18T11:58:57.686289+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:57.686289+0000","last_undegraded":"2024-09-18T11:58:57.686289+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36620091,"num_objects":8943,"num_object_clones":0,"num_object_copies":17886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":659,"num_objects_unfound":0,"num_objects_dirty":8943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10579,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2869,"num_bytes_recovered":11726848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8284},{"shards":"0,2","objects":659}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10426","reported_seq":10507,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.599717+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:57.599717+0000","last_peered":"2024-09-18T11:58:57.599717+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:57.599717+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35931346,"num_objects":8774,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10424,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5181},{"shards":"0,2","objects":2768}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10587","reported_seq":10672,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.688941+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:57.688941+0000","last_peered":"2024-09-18T11:58:57.688941+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:57.688941+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36559268,"num_objects":8929,"num_object_clones":0,"num_object_copies":17858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10584,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5406},{"shards":"0,2","objects":2697}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":583372876,"num_objects":142443,"num_object_clones":0,"num_object_copies":284886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":659,"num_objects_unfound":0,"num_objects_dirty":142443,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152772,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6369,"num_bytes_recovered":25985641,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1226346496,"data_stored":1226268772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160879,"ondisk_log_size":160879,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051645,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700936,"kb_used_data":398912,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93670904,"statfs":{"total":96636764160,"available":95919005696,"internally_reserved":0,"allocated":408485888,"data_stored":408258241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.377}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.253}]}]},{"osd":1,"up_from":35,"seq":150323855408,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626144,"kb_used_data":315800,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93745696,"statfs":{"total":96636764160,"available":95995592704,"internally_reserved":0,"allocated":323379200,"data_stored":323088788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.253},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":8.1899999999999995},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":2.8450000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":790592,"kb_used_data":484468,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93581248,"statfs":{"total":96636764160,"available":95827197952,"internally_reserved":0,"allocated":496095232,"data_stored":495868035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.247}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.38600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":495308800,"data_stored":495305321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323043328,"data_stored":322976634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407994368,"data_stored":407986817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:58:59.475 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:58:59.476 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'10548", 'reported_seq': 10619, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:57.680586+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:58:57.680586+0000', 'last_peered': '2024-09-18T11:58:57.680586+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:58:57.680586+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'500", 'ondisk_log_start': "30'500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36814848, 'num_objects': 8988, 'num_object_clones': 0, 'num_object_copies': 17976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10548, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5389}, {'shards': '0,2', 'objects': 2819}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:59.476 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:58:59.476 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'10595", 'reported_seq': 10778, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:57.979055+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:58:57.979055+0000', 'last_peered': '2024-09-18T11:58:57.979055+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:58:57.979055+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'500", 'ondisk_log_start': "30'500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36708376, 'num_objects': 8963, 'num_object_clones': 0, 'num_object_copies': 17926, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8963, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7453, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2721}, {'shards': '2', 'objects': 5427}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:59.476 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:58:59.476 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'10554", 'reported_seq': 10630, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:57.649267+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:58:57.649267+0000', 'last_peered': '2024-09-18T11:58:57.649267+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:58:57.649267+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'500", 'ondisk_log_start': "30'500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36328658, 'num_objects': 8871, 'num_object_clones': 0, 'num_object_copies': 17742, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10552, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5344}, {'shards': '0,2', 'objects': 2687}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:59.476 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:58:59.476 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'10468", 'reported_seq': 10554, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:57.653171+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:58:57.653171+0000', 'last_peered': '2024-09-18T11:58:57.653171+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:58:57.653171+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'400", 'ondisk_log_start': "30'400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36472018, 'num_objects': 8906, 'num_object_clones': 0, 'num_object_copies': 17812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10466, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5347}, {'shards': '0,2', 'objects': 2780}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:59.476 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:58:59.476 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'10566", 'reported_seq': 10641, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:57.681814+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:58:57.681814+0000', 'last_peered': '2024-09-18T11:58:57.681814+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:58:57.681814+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'500", 'ondisk_log_start': "30'500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36373714, 'num_objects': 8882, 'num_object_clones': 0, 'num_object_copies': 17764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10564, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5350}, {'shards': '0,2', 'objects': 2693}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:59.477 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:58:59.477 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'10573", 'reported_seq': 10629, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:58.004537+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:58:58.004537+0000', 'last_peered': '2024-09-18T11:58:58.004537+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:58:58.004537+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'500", 'ondisk_log_start': "30'500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36414674, 'num_objects': 8892, 'num_object_clones': 0, 'num_object_copies': 17784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10571, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2711}, {'shards': '2', 'objects': 5342}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:59.477 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:58:59.477 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'10567", 'reported_seq': 10776, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:57.994223+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:58:57.994223+0000', 'last_peered': '2024-09-18T11:58:57.994223+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:58:57.994223+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'500", 'ondisk_log_start': "30'500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36581376, 'num_objects': 8931, 'num_object_clones': 0, 'num_object_copies': 17862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7047, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2777}, {'shards': '2', 'objects': 5338}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:58:59.477 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:58:59.477 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'10614", 'reported_seq': 10696, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:57.675712+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:58:57.675712+0000', 'last_peered': '2024-09-18T11:58:57.675712+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:58:57.675712+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'600", 'ondisk_log_start': "30'600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36512361, 'num_objects': 8915, 'num_object_clones': 0, 'num_object_copies': 17830, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8915, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10613, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5380}, {'shards': '0,2', 'objects': 2686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:59.477 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:58:59.477 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'10582", 'reported_seq': 16399, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:58:57.686289+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:58:57.686289+0000', 'last_peered': '2024-09-18T11:58:57.686289+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:58:57.686289+0000', 'last_undegraded': '2024-09-18T11:58:57.686289+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "30'500", 'ondisk_log_start': "30'500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36620091, 'num_objects': 8943, 'num_object_clones': 0, 'num_object_copies': 17886, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 659, 'num_objects_unfound': 0, 'num_objects_dirty': 8943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10579, 'num_write_kb': 39032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2869, 'num_bytes_recovered': 11726848, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 8284}, {'shards': '0,2', 'objects': 659}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:59.477 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:58:59.477 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'10426", 'reported_seq': 10507, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:57.599717+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:58:57.599717+0000', 'last_peered': '2024-09-18T11:58:57.599717+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:58:57.599717+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'400", 'ondisk_log_start': "30'400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35931346, 'num_objects': 8774, 'num_object_clones': 0, 'num_object_copies': 17548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10424, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5181}, {'shards': '0,2', 'objects': 2768}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:59.478 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:58:59.478 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'10587", 'reported_seq': 10672, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:58:57.688941+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:58:57.688941+0000', 'last_peered': '2024-09-18T11:58:57.688941+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:58:57.688941+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'500", 'ondisk_log_start': "30'500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36559268, 'num_objects': 8929, 'num_object_clones': 0, 'num_object_copies': 17858, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8929, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10584, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5406}, {'shards': '0,2', 'objects': 2697}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:58:59.478 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:59:00.422 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:00.422 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/284890 objects degraded (12.483%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":340,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":142445,"data_bytes":583372895,"bytes_used":2168496128,"bytes_avail":287741796352,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":284890,"degraded_ratio":0.12483414651268911,"misplaced_objects":659,"misplaced_total":284890,"misplaced_ratio":0.0023131735055635511,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":154532,"recovering_keys_per_sec":0,"num_objects_recovered":453,"num_bytes_recovered":1855488,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":520},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:59:00.464 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:59:00.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:01.120 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:01.141 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-18T11:58:59.127615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583372895,"num_objects":142445,"num_object_clones":0,"num_object_copies":284890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":659,"num_objects_unfound":0,"num_objects_dirty":142445,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152774,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6370,"num_bytes_recovered":25985660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2117672,"kb_used_data":1199180,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":280997848,"statfs":{"total":289910292480,"available":287741796352,"internally_reserved":0,"allocated":1227960320,"data_stored":1227215064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,5,6,11,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25595904,"num_objects":-6249,"num_object_clones":0,"num_object_copies":-12498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-453,"num_objects_unfound":0,"num_objects_dirty":-6249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6249,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":453,"num_bytes_recovered":1855488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007142"},"pg_stats":[{"pgid":"2.f","version":"39'10548","reported_seq":10619,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.680586+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:58:57.680586+0000","last_peered":"2024-09-18T11:58:57.680586+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:58:57.680586+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36814848,"num_objects":8988,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10548,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5389},{"shards":"0,2","objects":2819}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10546","reported_seq":10700,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:58.023039+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:58:58.023039+0000","last_peered":"2024-09-18T11:58:58.023039+0000","last_clean":"2024-09-18T11:58:58.023039+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:58:58.023039+0000","last_undegraded":"2024-09-18T11:58:58.023039+0000","last_fullsized":"2024-09-18T11:58:58.023039+0000","mapping_epoch":29,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36397056,"num_objects":8886,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7738,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10595","reported_seq":10778,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.979055+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:58:57.979055+0000","last_peered":"2024-09-18T11:58:57.979055+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:58:57.979055+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36708376,"num_objects":8963,"num_object_clones":0,"num_object_copies":17926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8963,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7453,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2721},{"shards":"2","objects":5427}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10449","reported_seq":10478,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:57.791166+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:58:57.791166+0000","last_peered":"2024-09-18T11:58:57.791166+0000","last_clean":"2024-09-18T11:58:57.791166+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:58:57.791166+0000","last_undegraded":"2024-09-18T11:58:57.791166+0000","last_fullsized":"2024-09-18T11:58:57.791166+0000","mapping_epoch":29,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36089856,"num_objects":8811,"num_object_clones":0,"num_object_copies":17622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10449,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10523","reported_seq":10714,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:57.648970+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:58:57.648970+0000","last_peered":"2024-09-18T11:58:57.648970+0000","last_clean":"2024-09-18T11:58:57.648970+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:58:57.648970+0000","last_undegraded":"2024-09-18T11:58:57.648970+0000","last_fullsized":"2024-09-18T11:58:57.648970+0000","mapping_epoch":29,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36552704,"num_objects":8924,"num_object_clones":0,"num_object_copies":17848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7688,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10554","reported_seq":10630,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.649267+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:58:57.649267+0000","last_peered":"2024-09-18T11:58:57.649267+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:58:57.649267+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36328658,"num_objects":8871,"num_object_clones":0,"num_object_copies":17742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10552,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5344},{"shards":"0,2","objects":2687}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10468","reported_seq":10554,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.653171+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:58:57.653171+0000","last_peered":"2024-09-18T11:58:57.653171+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:58:57.653171+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36472018,"num_objects":8906,"num_object_clones":0,"num_object_copies":17812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10466,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5347},{"shards":"0,2","objects":2780}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10655","reported_seq":10880,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:57.656446+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:58:57.656446+0000","last_peered":"2024-09-18T11:58:57.656446+0000","last_clean":"2024-09-18T11:58:57.656446+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:58:57.656446+0000","last_undegraded":"2024-09-18T11:58:57.656446+0000","last_fullsized":"2024-09-18T11:58:57.656446+0000","mapping_epoch":29,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36646912,"num_objects":8947,"num_object_clones":0,"num_object_copies":17894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6972,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10526","reported_seq":17654,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:58:57.971986+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:58:57.971986+0000","last_peered":"2024-09-18T11:58:57.971986+0000","last_clean":"2024-09-18T11:58:57.971986+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:58:57.971986+0000","last_undegraded":"2024-09-18T11:58:57.971986+0000","last_fullsized":"2024-09-18T11:58:57.971986+0000","mapping_epoch":38,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36369618,"num_objects":8881,"num_object_clones":0,"num_object_copies":17762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10524,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10566","reported_seq":10641,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.681814+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:58:57.681814+0000","last_peered":"2024-09-18T11:58:57.681814+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:58:57.681814+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36373714,"num_objects":8882,"num_object_clones":0,"num_object_copies":17764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10564,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5350},{"shards":"0,2","objects":2693}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10573","reported_seq":10629,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:58.004537+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:58:58.004537+0000","last_peered":"2024-09-18T11:58:58.004537+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:58:58.004537+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36414674,"num_objects":8892,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10571,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2711},{"shards":"2","objects":5342}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10567","reported_seq":10776,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.994223+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:58:57.994223+0000","last_peered":"2024-09-18T11:58:57.994223+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:58:57.994223+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36581376,"num_objects":8931,"num_object_clones":0,"num_object_copies":17862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7047,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2777},{"shards":"2","objects":5338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10614","reported_seq":10696,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.675712+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:58:57.675712+0000","last_peered":"2024-09-18T11:58:57.675712+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:58:57.675712+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36512361,"num_objects":8915,"num_object_clones":0,"num_object_copies":17830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5380},{"shards":"0,2","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10582","reported_seq":16399,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:58:57.686289+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:58:57.686289+0000","last_peered":"2024-09-18T11:58:57.686289+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:58:57.686289+0000","last_undegraded":"2024-09-18T11:58:57.686289+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36620091,"num_objects":8943,"num_object_clones":0,"num_object_copies":17886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":659,"num_objects_unfound":0,"num_objects_dirty":8943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10579,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2869,"num_bytes_recovered":11726848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8284},{"shards":"0,2","objects":659}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10426","reported_seq":10507,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.599717+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:58:57.599717+0000","last_peered":"2024-09-18T11:58:57.599717+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:58:57.599717+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'400","ondisk_log_start":"30'400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35931346,"num_objects":8774,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10424,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5181},{"shards":"0,2","objects":2768}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10587","reported_seq":10672,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:58:57.688941+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:58:57.688941+0000","last_peered":"2024-09-18T11:58:57.688941+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:58:57.688941+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36559268,"num_objects":8929,"num_object_clones":0,"num_object_copies":17858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10584,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5406},{"shards":"0,2","objects":2697}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":583372876,"num_objects":142443,"num_object_clones":0,"num_object_copies":284886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":659,"num_objects_unfound":0,"num_objects_dirty":142443,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152772,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6369,"num_bytes_recovered":25985641,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1226346496,"data_stored":1226268772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160879,"ondisk_log_size":160879,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051645,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700936,"kb_used_data":398912,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93670904,"statfs":{"total":96636764160,"available":95919005696,"internally_reserved":0,"allocated":408485888,"data_stored":408258241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.377}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.253}]}]},{"osd":1,"up_from":35,"seq":150323855408,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626144,"kb_used_data":315800,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93745696,"statfs":{"total":96636764160,"available":95995592704,"internally_reserved":0,"allocated":323379200,"data_stored":323088788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.253},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":8.1899999999999995},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":2.8450000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":790592,"kb_used_data":484468,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93581248,"statfs":{"total":96636764160,"available":95827197952,"internally_reserved":0,"allocated":496095232,"data_stored":495868035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.247}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.38600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":495308800,"data_stored":495305321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":323043328,"data_stored":322976634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407994368,"data_stored":407986817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:04.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:05.052 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:05.090 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-18T11:59:03.131502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":574955615,"num_objects":140390,"num_object_clones":0,"num_object_copies":280780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":477,"num_objects_unfound":0,"num_objects_dirty":140390,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154829,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6552,"num_bytes_recovered":26731132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160837,"ondisk_log_size":160837,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2102964,"kb_used_data":1188628,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281012556,"statfs":{"total":289910292480,"available":287756857344,"internally_reserved":0,"allocated":1217155072,"data_stored":1216409816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,7,4,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25505792,"num_objects":-6227,"num_object_clones":0,"num_object_copies":-12454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-490,"num_objects_unfound":0,"num_objects_dirty":-6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":490,"num_bytes_recovered":2007040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010140"},"pg_stats":[{"pgid":"2.f","version":"39'10673","reported_seq":10744,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.630910+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:02.630910+0000","last_peered":"2024-09-18T11:59:02.630910+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:02.630910+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36302848,"num_objects":8863,"num_object_clones":0,"num_object_copies":17726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10673,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5264},{"shards":"0,2","objects":2694}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10665","reported_seq":10823,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.900139+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:02.900139+0000","last_peered":"2024-09-18T11:59:02.900139+0000","last_clean":"2024-09-18T11:59:02.900139+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:02.900139+0000","last_undegraded":"2024-09-18T11:59:02.900139+0000","last_fullsized":"2024-09-18T11:59:02.900139+0000","mapping_epoch":29,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35909632,"num_objects":8767,"num_object_clones":0,"num_object_copies":17534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10725","reported_seq":10911,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.951257+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:02.951257+0000","last_peered":"2024-09-18T11:59:02.951257+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:02.951257+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36175896,"num_objects":8833,"num_object_clones":0,"num_object_copies":17666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8833,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7583,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2591},{"shards":"2","objects":5297}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10585","reported_seq":10614,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.992966+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:02.992966+0000","last_peered":"2024-09-18T11:59:02.992966+0000","last_clean":"2024-09-18T11:59:02.992966+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:02.992966+0000","last_undegraded":"2024-09-18T11:59:02.992966+0000","last_fullsized":"2024-09-18T11:59:02.992966+0000","mapping_epoch":29,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35532800,"num_objects":8675,"num_object_clones":0,"num_object_copies":17350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10585,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10635","reported_seq":10826,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.682388+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:02.682388+0000","last_peered":"2024-09-18T11:59:02.682388+0000","last_clean":"2024-09-18T11:59:02.682388+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:02.682388+0000","last_undegraded":"2024-09-18T11:59:02.682388+0000","last_fullsized":"2024-09-18T11:59:02.682388+0000","mapping_epoch":29,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36093952,"num_objects":8812,"num_object_clones":0,"num_object_copies":17624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10705","reported_seq":10785,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.602756+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:02.602756+0000","last_peered":"2024-09-18T11:59:02.602756+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:02.602756+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35710162,"num_objects":8720,"num_object_clones":0,"num_object_copies":17440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10703,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5193},{"shards":"0,2","objects":2536}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10602","reported_seq":10689,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.680888+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:02.680888+0000","last_peered":"2024-09-18T11:59:02.680888+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:02.680888+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35923154,"num_objects":8772,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5213},{"shards":"0,2","objects":2646}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10794","reported_seq":11019,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.611862+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:02.611862+0000","last_peered":"2024-09-18T11:59:02.611862+0000","last_clean":"2024-09-18T11:59:02.611862+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:02.611862+0000","last_undegraded":"2024-09-18T11:59:02.611862+0000","last_fullsized":"2024-09-18T11:59:02.611862+0000","mapping_epoch":29,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36077568,"num_objects":8808,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7111,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10665","reported_seq":17793,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.982463+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:02.982463+0000","last_peered":"2024-09-18T11:59:02.982463+0000","last_clean":"2024-09-18T11:59:02.982463+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:02.982463+0000","last_undegraded":"2024-09-18T11:59:02.982463+0000","last_fullsized":"2024-09-18T11:59:02.982463+0000","mapping_epoch":38,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35800274,"num_objects":8742,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10670","reported_seq":10745,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.660001+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:02.660001+0000","last_peered":"2024-09-18T11:59:02.660001+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:02.660001+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35947730,"num_objects":8778,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10668,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5246},{"shards":"0,2","objects":2589}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10684","reported_seq":10740,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:03.020650+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:03.020650+0000","last_peered":"2024-09-18T11:59:03.020650+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:03.020650+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35960018,"num_objects":8781,"num_object_clones":0,"num_object_copies":17562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10682,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2600},{"shards":"2","objects":5231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10705","reported_seq":10912,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:03.019282+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:59:03.019282+0000","last_peered":"2024-09-18T11:59:03.019282+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:03.019282+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36016128,"num_objects":8793,"num_object_clones":0,"num_object_copies":17586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7185,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2639},{"shards":"2","objects":5200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10744","reported_seq":10825,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.683397+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:02.683397+0000","last_peered":"2024-09-18T11:59:02.683397+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:02.683397+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35979881,"num_objects":8785,"num_object_clones":0,"num_object_copies":17570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10743,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5250},{"shards":"0,2","objects":2556}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10700","reported_seq":16881,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:59:02.731043+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:59:02.731043+0000","last_peered":"2024-09-18T11:59:02.731043+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:59:02.731043+0000","last_undegraded":"2024-09-18T11:59:02.731043+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36136763,"num_objects":8825,"num_object_clones":0,"num_object_copies":17650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":477,"num_objects_unfound":0,"num_objects_dirty":8825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10697,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3051,"num_bytes_recovered":12472320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8348},{"shards":"0,2","objects":477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10557","reported_seq":10638,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.715758+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:02.715758+0000","last_peered":"2024-09-18T11:59:02.715758+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:02.715758+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35394770,"num_objects":8643,"num_object_clones":0,"num_object_copies":17286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10555,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5050},{"shards":"0,2","objects":2637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10725","reported_seq":10810,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.645350+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:02.645350+0000","last_peered":"2024-09-18T11:59:02.645350+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:02.645350+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35994020,"num_objects":8791,"num_object_clones":0,"num_object_copies":17582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5268},{"shards":"0,2","objects":2559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":574955596,"num_objects":140388,"num_object_clones":0,"num_object_copies":280776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":477,"num_objects_unfound":0,"num_objects_dirty":140388,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154827,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6551,"num_bytes_recovered":26731113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1215488000,"data_stored":1215410276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160834,"ondisk_log_size":160834,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051646,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700976,"kb_used_data":394792,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93670864,"statfs":{"total":96636764160,"available":95918964736,"internally_reserved":0,"allocated":404267008,"data_stored":404039361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.184},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.29699999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":3.1139999999999999},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.218}]}]},{"osd":1,"up_from":35,"seq":150323855409,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613444,"kb_used_data":315580,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93758396,"statfs":{"total":96636764160,"available":96008597504,"internally_reserved":0,"allocated":323153920,"data_stored":322863508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.377},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.40300000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788544,"kb_used_data":478256,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93583296,"statfs":{"total":96636764160,"available":95829295104,"internally_reserved":0,"allocated":489734144,"data_stored":489506947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":3.9500000000000002},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":3.9329999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":6.0419999999999998},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.23499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":488976384,"data_stored":488972905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322764800,"data_stored":322698106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":403746816,"data_stored":403739265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:05.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:05.883 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:05.905 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:05.911 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-18T11:59:05.132021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":574955615,"num_objects":140390,"num_object_clones":0,"num_object_copies":280780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":477,"num_objects_unfound":0,"num_objects_dirty":140390,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154829,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6552,"num_bytes_recovered":26731132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160837,"ondisk_log_size":160837,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2102964,"kb_used_data":1188628,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281012556,"statfs":{"total":289910292480,"available":287756857344,"internally_reserved":0,"allocated":1217155072,"data_stored":1216409816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,7,4,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17027072,"num_objects":-4157,"num_object_clones":0,"num_object_copies":-8314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-328,"num_objects_unfound":0,"num_objects_dirty":-4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4157,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010231"},"pg_stats":[{"pgid":"2.f","version":"39'10673","reported_seq":10744,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.630910+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:02.630910+0000","last_peered":"2024-09-18T11:59:02.630910+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:02.630910+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36302848,"num_objects":8863,"num_object_clones":0,"num_object_copies":17726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10673,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5264},{"shards":"0,2","objects":2694}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10665","reported_seq":10823,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.900139+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:02.900139+0000","last_peered":"2024-09-18T11:59:02.900139+0000","last_clean":"2024-09-18T11:59:02.900139+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:02.900139+0000","last_undegraded":"2024-09-18T11:59:02.900139+0000","last_fullsized":"2024-09-18T11:59:02.900139+0000","mapping_epoch":29,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35909632,"num_objects":8767,"num_object_clones":0,"num_object_copies":17534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10725","reported_seq":10911,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.951257+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:02.951257+0000","last_peered":"2024-09-18T11:59:02.951257+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:02.951257+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36175896,"num_objects":8833,"num_object_clones":0,"num_object_copies":17666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8833,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7583,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2591},{"shards":"2","objects":5297}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10585","reported_seq":10614,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.992966+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:02.992966+0000","last_peered":"2024-09-18T11:59:02.992966+0000","last_clean":"2024-09-18T11:59:02.992966+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:02.992966+0000","last_undegraded":"2024-09-18T11:59:02.992966+0000","last_fullsized":"2024-09-18T11:59:02.992966+0000","mapping_epoch":29,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35532800,"num_objects":8675,"num_object_clones":0,"num_object_copies":17350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10585,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10635","reported_seq":10826,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.682388+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:02.682388+0000","last_peered":"2024-09-18T11:59:02.682388+0000","last_clean":"2024-09-18T11:59:02.682388+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:02.682388+0000","last_undegraded":"2024-09-18T11:59:02.682388+0000","last_fullsized":"2024-09-18T11:59:02.682388+0000","mapping_epoch":29,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36093952,"num_objects":8812,"num_object_clones":0,"num_object_copies":17624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10705","reported_seq":10785,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.602756+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:02.602756+0000","last_peered":"2024-09-18T11:59:02.602756+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:02.602756+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35710162,"num_objects":8720,"num_object_clones":0,"num_object_copies":17440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10703,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5193},{"shards":"0,2","objects":2536}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10602","reported_seq":10689,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.680888+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:02.680888+0000","last_peered":"2024-09-18T11:59:02.680888+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:02.680888+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35923154,"num_objects":8772,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5213},{"shards":"0,2","objects":2646}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10794","reported_seq":11019,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.611862+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:02.611862+0000","last_peered":"2024-09-18T11:59:02.611862+0000","last_clean":"2024-09-18T11:59:02.611862+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:02.611862+0000","last_undegraded":"2024-09-18T11:59:02.611862+0000","last_fullsized":"2024-09-18T11:59:02.611862+0000","mapping_epoch":29,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36077568,"num_objects":8808,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7111,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10665","reported_seq":17793,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.982463+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:02.982463+0000","last_peered":"2024-09-18T11:59:02.982463+0000","last_clean":"2024-09-18T11:59:02.982463+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:02.982463+0000","last_undegraded":"2024-09-18T11:59:02.982463+0000","last_fullsized":"2024-09-18T11:59:02.982463+0000","mapping_epoch":38,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35800274,"num_objects":8742,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10670","reported_seq":10745,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.660001+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:02.660001+0000","last_peered":"2024-09-18T11:59:02.660001+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:02.660001+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35947730,"num_objects":8778,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10668,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5246},{"shards":"0,2","objects":2589}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10684","reported_seq":10740,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:03.020650+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:03.020650+0000","last_peered":"2024-09-18T11:59:03.020650+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:03.020650+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35960018,"num_objects":8781,"num_object_clones":0,"num_object_copies":17562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10682,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2600},{"shards":"2","objects":5231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10705","reported_seq":10912,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:03.019282+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:59:03.019282+0000","last_peered":"2024-09-18T11:59:03.019282+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:03.019282+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36016128,"num_objects":8793,"num_object_clones":0,"num_object_copies":17586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7185,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2639},{"shards":"2","objects":5200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10744","reported_seq":10825,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.683397+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:02.683397+0000","last_peered":"2024-09-18T11:59:02.683397+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:02.683397+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35979881,"num_objects":8785,"num_object_clones":0,"num_object_copies":17570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10743,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5250},{"shards":"0,2","objects":2556}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10700","reported_seq":16881,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:59:02.731043+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:59:02.731043+0000","last_peered":"2024-09-18T11:59:02.731043+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:59:02.731043+0000","last_undegraded":"2024-09-18T11:59:02.731043+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36136763,"num_objects":8825,"num_object_clones":0,"num_object_copies":17650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":477,"num_objects_unfound":0,"num_objects_dirty":8825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10697,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3051,"num_bytes_recovered":12472320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8348},{"shards":"0,2","objects":477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10557","reported_seq":10638,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.715758+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:02.715758+0000","last_peered":"2024-09-18T11:59:02.715758+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:02.715758+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35394770,"num_objects":8643,"num_object_clones":0,"num_object_copies":17286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10555,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5050},{"shards":"0,2","objects":2637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10725","reported_seq":10810,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.645350+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:02.645350+0000","last_peered":"2024-09-18T11:59:02.645350+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:02.645350+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35994020,"num_objects":8791,"num_object_clones":0,"num_object_copies":17582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5268},{"shards":"0,2","objects":2559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":574955596,"num_objects":140388,"num_object_clones":0,"num_object_copies":280776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":477,"num_objects_unfound":0,"num_objects_dirty":140388,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154827,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6551,"num_bytes_recovered":26731113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1215488000,"data_stored":1215410276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160834,"ondisk_log_size":160834,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051646,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700976,"kb_used_data":394792,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93670864,"statfs":{"total":96636764160,"available":95918964736,"internally_reserved":0,"allocated":404267008,"data_stored":404039361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.184},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.29699999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":3.1139999999999999},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.218}]}]},{"osd":1,"up_from":35,"seq":150323855409,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613444,"kb_used_data":315580,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93758396,"statfs":{"total":96636764160,"available":96008597504,"internally_reserved":0,"allocated":323153920,"data_stored":322863508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.377},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.40300000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788544,"kb_used_data":478256,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93583296,"statfs":{"total":96636764160,"available":95829295104,"internally_reserved":0,"allocated":489734144,"data_stored":489506947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":3.9500000000000002},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":3.9329999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":6.0419999999999998},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.23499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":488976384,"data_stored":488972905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322764800,"data_stored":322698106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":403746816,"data_stored":403739265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:05.912 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:59:05.912 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'10673", 'reported_seq': 10744, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:02.630910+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:59:02.630910+0000', 'last_peered': '2024-09-18T11:59:02.630910+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:59:02.630910+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'600", 'ondisk_log_start': "30'600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36302848, 'num_objects': 8863, 'num_object_clones': 0, 'num_object_copies': 17726, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10673, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5264}, {'shards': '0,2', 'objects': 2694}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:05.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:59:05.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'10725", 'reported_seq': 10911, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:02.951257+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:59:02.951257+0000', 'last_peered': '2024-09-18T11:59:02.951257+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:59:02.951257+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'700", 'ondisk_log_start': "30'700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36175896, 'num_objects': 8833, 'num_object_clones': 0, 'num_object_copies': 17666, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8833, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7583, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2591}, {'shards': '2', 'objects': 5297}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:05.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:59:05.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'10705", 'reported_seq': 10785, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:02.602756+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:59:02.602756+0000', 'last_peered': '2024-09-18T11:59:02.602756+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:59:02.602756+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'700", 'ondisk_log_start': "30'700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35710162, 'num_objects': 8720, 'num_object_clones': 0, 'num_object_copies': 17440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10703, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5193}, {'shards': '0,2', 'objects': 2536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:05.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:59:05.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'10602", 'reported_seq': 10689, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:02.680888+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:59:02.680888+0000', 'last_peered': '2024-09-18T11:59:02.680888+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:59:02.680888+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'600", 'ondisk_log_start': "30'600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35923154, 'num_objects': 8772, 'num_object_clones': 0, 'num_object_copies': 17544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10600, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5213}, {'shards': '0,2', 'objects': 2646}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:05.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:59:05.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'10670", 'reported_seq': 10745, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:02.660001+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:59:02.660001+0000', 'last_peered': '2024-09-18T11:59:02.660001+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:59:02.660001+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'600", 'ondisk_log_start': "30'600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35947730, 'num_objects': 8778, 'num_object_clones': 0, 'num_object_copies': 17556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10668, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5246}, {'shards': '0,2', 'objects': 2589}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:05.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:59:05.913 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'10684", 'reported_seq': 10740, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:03.020650+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:59:03.020650+0000', 'last_peered': '2024-09-18T11:59:03.020650+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:59:03.020650+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'600", 'ondisk_log_start': "30'600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35960018, 'num_objects': 8781, 'num_object_clones': 0, 'num_object_copies': 17562, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8781, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10682, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2600}, {'shards': '2', 'objects': 5231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:05.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:59:05.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'10705", 'reported_seq': 10912, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:03.019282+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:59:03.019282+0000', 'last_peered': '2024-09-18T11:59:03.019282+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:59:03.019282+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'700", 'ondisk_log_start': "30'700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36016128, 'num_objects': 8793, 'num_object_clones': 0, 'num_object_copies': 17586, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7185, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2639}, {'shards': '2', 'objects': 5200}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:05.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:59:05.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'10744", 'reported_seq': 10825, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:02.683397+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:59:02.683397+0000', 'last_peered': '2024-09-18T11:59:02.683397+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:59:02.683397+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'700", 'ondisk_log_start': "30'700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35979881, 'num_objects': 8785, 'num_object_clones': 0, 'num_object_copies': 17570, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8785, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10743, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5250}, {'shards': '0,2', 'objects': 2556}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:05.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:59:05.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'10700", 'reported_seq': 16881, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:59:02.731043+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:59:02.731043+0000', 'last_peered': '2024-09-18T11:59:02.731043+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:59:02.731043+0000', 'last_undegraded': '2024-09-18T11:59:02.731043+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "30'600", 'ondisk_log_start': "30'600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36136763, 'num_objects': 8825, 'num_object_clones': 0, 'num_object_copies': 17650, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 477, 'num_objects_unfound': 0, 'num_objects_dirty': 8825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10697, 'num_write_kb': 39032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3051, 'num_bytes_recovered': 12472320, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 8348}, {'shards': '0,2', 'objects': 477}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:05.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:59:05.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'10557", 'reported_seq': 10638, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:02.715758+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:59:02.715758+0000', 'last_peered': '2024-09-18T11:59:02.715758+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:59:02.715758+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'500", 'ondisk_log_start': "30'500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35394770, 'num_objects': 8643, 'num_object_clones': 0, 'num_object_copies': 17286, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8643, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10555, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5050}, {'shards': '0,2', 'objects': 2637}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:05.914 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:59:05.915 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'10725", 'reported_seq': 10810, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:02.645350+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:59:02.645350+0000', 'last_peered': '2024-09-18T11:59:02.645350+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:59:02.645350+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'700", 'ondisk_log_start': "30'700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35994020, 'num_objects': 8791, 'num_object_clones': 0, 'num_object_copies': 17582, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10722, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5268}, {'shards': '0,2', 'objects': 2559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:05.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:59:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:06.823 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/280780 objects degraded (12.666%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":346,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":140390,"data_bytes":574955615,"bytes_used":2153435136,"bytes_avail":287756857344,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":280780,"degraded_ratio":0.12666144312272953,"misplaced_objects":477,"misplaced_total":280780,"misplaced_ratio":0.0016988389486430658,"recovering_objects_per_sec":27,"recovering_bytes_per_sec":111861,"recovering_keys_per_sec":0,"num_objects_recovered":328,"num_bytes_recovered":1343488,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":346},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:59:06.846 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:59:06.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:07.528 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:07.538 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:07.548 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-09-18T11:59:07.132468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":574955615,"num_objects":140390,"num_object_clones":0,"num_object_copies":280780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":477,"num_objects_unfound":0,"num_objects_dirty":140390,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154829,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6552,"num_bytes_recovered":26731132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160837,"ondisk_log_size":160837,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2102964,"kb_used_data":1188628,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281012556,"statfs":{"total":289910292480,"available":287756857344,"internally_reserved":0,"allocated":1217155072,"data_stored":1216409816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,7,4,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17027072,"num_objects":-4157,"num_object_clones":0,"num_object_copies":-8314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-328,"num_objects_unfound":0,"num_objects_dirty":-4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4157,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010189"},"pg_stats":[{"pgid":"2.f","version":"39'10673","reported_seq":10744,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.630910+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:02.630910+0000","last_peered":"2024-09-18T11:59:02.630910+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:02.630910+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36302848,"num_objects":8863,"num_object_clones":0,"num_object_copies":17726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10673,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5264},{"shards":"0,2","objects":2694}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10665","reported_seq":10823,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.900139+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:02.900139+0000","last_peered":"2024-09-18T11:59:02.900139+0000","last_clean":"2024-09-18T11:59:02.900139+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:02.900139+0000","last_undegraded":"2024-09-18T11:59:02.900139+0000","last_fullsized":"2024-09-18T11:59:02.900139+0000","mapping_epoch":29,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35909632,"num_objects":8767,"num_object_clones":0,"num_object_copies":17534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10725","reported_seq":10911,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.951257+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:02.951257+0000","last_peered":"2024-09-18T11:59:02.951257+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:02.951257+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36175896,"num_objects":8833,"num_object_clones":0,"num_object_copies":17666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8833,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7583,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2591},{"shards":"2","objects":5297}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10585","reported_seq":10614,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.992966+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:02.992966+0000","last_peered":"2024-09-18T11:59:02.992966+0000","last_clean":"2024-09-18T11:59:02.992966+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:02.992966+0000","last_undegraded":"2024-09-18T11:59:02.992966+0000","last_fullsized":"2024-09-18T11:59:02.992966+0000","mapping_epoch":29,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35532800,"num_objects":8675,"num_object_clones":0,"num_object_copies":17350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10585,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10635","reported_seq":10826,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.682388+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:02.682388+0000","last_peered":"2024-09-18T11:59:02.682388+0000","last_clean":"2024-09-18T11:59:02.682388+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:02.682388+0000","last_undegraded":"2024-09-18T11:59:02.682388+0000","last_fullsized":"2024-09-18T11:59:02.682388+0000","mapping_epoch":29,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36093952,"num_objects":8812,"num_object_clones":0,"num_object_copies":17624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10705","reported_seq":10785,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.602756+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:02.602756+0000","last_peered":"2024-09-18T11:59:02.602756+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:02.602756+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35710162,"num_objects":8720,"num_object_clones":0,"num_object_copies":17440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10703,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5193},{"shards":"0,2","objects":2536}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10602","reported_seq":10689,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.680888+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:02.680888+0000","last_peered":"2024-09-18T11:59:02.680888+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:02.680888+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35923154,"num_objects":8772,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5213},{"shards":"0,2","objects":2646}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10794","reported_seq":11019,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.611862+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:02.611862+0000","last_peered":"2024-09-18T11:59:02.611862+0000","last_clean":"2024-09-18T11:59:02.611862+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:02.611862+0000","last_undegraded":"2024-09-18T11:59:02.611862+0000","last_fullsized":"2024-09-18T11:59:02.611862+0000","mapping_epoch":29,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36077568,"num_objects":8808,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7111,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10665","reported_seq":17793,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:02.982463+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:02.982463+0000","last_peered":"2024-09-18T11:59:02.982463+0000","last_clean":"2024-09-18T11:59:02.982463+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:02.982463+0000","last_undegraded":"2024-09-18T11:59:02.982463+0000","last_fullsized":"2024-09-18T11:59:02.982463+0000","mapping_epoch":38,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35800274,"num_objects":8742,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10670","reported_seq":10745,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.660001+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:02.660001+0000","last_peered":"2024-09-18T11:59:02.660001+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:02.660001+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35947730,"num_objects":8778,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10668,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5246},{"shards":"0,2","objects":2589}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10684","reported_seq":10740,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:03.020650+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:03.020650+0000","last_peered":"2024-09-18T11:59:03.020650+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:03.020650+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35960018,"num_objects":8781,"num_object_clones":0,"num_object_copies":17562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10682,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2600},{"shards":"2","objects":5231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10705","reported_seq":10912,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:03.019282+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:59:03.019282+0000","last_peered":"2024-09-18T11:59:03.019282+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:03.019282+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36016128,"num_objects":8793,"num_object_clones":0,"num_object_copies":17586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7185,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2639},{"shards":"2","objects":5200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10744","reported_seq":10825,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.683397+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:02.683397+0000","last_peered":"2024-09-18T11:59:02.683397+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:02.683397+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35979881,"num_objects":8785,"num_object_clones":0,"num_object_copies":17570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10743,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5250},{"shards":"0,2","objects":2556}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10700","reported_seq":16881,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:59:02.731043+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:59:02.731043+0000","last_peered":"2024-09-18T11:59:02.731043+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:59:02.731043+0000","last_undegraded":"2024-09-18T11:59:02.731043+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36136763,"num_objects":8825,"num_object_clones":0,"num_object_copies":17650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":477,"num_objects_unfound":0,"num_objects_dirty":8825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10697,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3051,"num_bytes_recovered":12472320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8348},{"shards":"0,2","objects":477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10557","reported_seq":10638,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.715758+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:02.715758+0000","last_peered":"2024-09-18T11:59:02.715758+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:02.715758+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'500","ondisk_log_start":"30'500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35394770,"num_objects":8643,"num_object_clones":0,"num_object_copies":17286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10555,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5050},{"shards":"0,2","objects":2637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10725","reported_seq":10810,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:02.645350+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:02.645350+0000","last_peered":"2024-09-18T11:59:02.645350+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:02.645350+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35994020,"num_objects":8791,"num_object_clones":0,"num_object_copies":17582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5268},{"shards":"0,2","objects":2559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":574955596,"num_objects":140388,"num_object_clones":0,"num_object_copies":280776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":477,"num_objects_unfound":0,"num_objects_dirty":140388,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154827,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6551,"num_bytes_recovered":26731113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1215488000,"data_stored":1215410276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160834,"ondisk_log_size":160834,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051646,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700976,"kb_used_data":394792,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93670864,"statfs":{"total":96636764160,"available":95918964736,"internally_reserved":0,"allocated":404267008,"data_stored":404039361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.184},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.29699999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":3.1139999999999999},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.218}]}]},{"osd":1,"up_from":35,"seq":150323855409,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613444,"kb_used_data":315580,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93758396,"statfs":{"total":96636764160,"available":96008597504,"internally_reserved":0,"allocated":323153920,"data_stored":322863508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.377},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.40300000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788544,"kb_used_data":478256,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93583296,"statfs":{"total":96636764160,"available":95829295104,"internally_reserved":0,"allocated":489734144,"data_stored":489506947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":3.9500000000000002},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":3.9329999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":6.0419999999999998},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.23499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":488976384,"data_stored":488972905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322764800,"data_stored":322698106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":403746816,"data_stored":403739265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:10.550 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:11.237 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:11.294 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:11.301 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-18T11:59:11.133405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":566435935,"num_objects":138310,"num_object_clones":0,"num_object_copies":276620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":289,"num_objects_unfound":0,"num_objects_dirty":138310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156909,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6740,"num_bytes_recovered":27501180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160717,"ondisk_log_size":160717,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2092260,"kb_used_data":1177928,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281023260,"statfs":{"total":289910292480,"available":287767818240,"internally_reserved":0,"allocated":1206198272,"data_stored":1205453016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,9,9,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16936960,"num_objects":-4135,"num_object_clones":0,"num_object_copies":-8270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-370,"num_objects_unfound":0,"num_objects_dirty":-4135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4135,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":370,"num_bytes_recovered":1515520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005790"},"pg_stats":[{"pgid":"2.f","version":"39'10807","reported_seq":10878,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.683024+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:07.683024+0000","last_peered":"2024-09-18T11:59:07.683024+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:07.683024+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35753984,"num_objects":8729,"num_object_clones":0,"num_object_copies":17458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5130},{"shards":"0,2","objects":2560}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10799","reported_seq":10955,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:08.060881+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:08.060881+0000","last_peered":"2024-09-18T11:59:08.060881+0000","last_clean":"2024-09-18T11:59:08.060881+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:08.060881+0000","last_undegraded":"2024-09-18T11:59:08.060881+0000","last_fullsized":"2024-09-18T11:59:08.060881+0000","mapping_epoch":29,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35360768,"num_objects":8633,"num_object_clones":0,"num_object_copies":17266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7991,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10860","reported_seq":11046,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:08.079539+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:08.079539+0000","last_peered":"2024-09-18T11:59:08.079539+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:08.079539+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35622936,"num_objects":8698,"num_object_clones":0,"num_object_copies":17396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8698,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7718,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2456},{"shards":"2","objects":5162}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10711","reported_seq":10740,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:08.025504+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:08.025504+0000","last_peered":"2024-09-18T11:59:08.025504+0000","last_clean":"2024-09-18T11:59:08.025504+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:08.025504+0000","last_undegraded":"2024-09-18T11:59:08.025504+0000","last_fullsized":"2024-09-18T11:59:08.025504+0000","mapping_epoch":29,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35016704,"num_objects":8549,"num_object_clones":0,"num_object_copies":17098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10711,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10770","reported_seq":10961,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:07.721292+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:07.721292+0000","last_peered":"2024-09-18T11:59:07.721292+0000","last_clean":"2024-09-18T11:59:07.721292+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:07.721292+0000","last_undegraded":"2024-09-18T11:59:07.721292+0000","last_fullsized":"2024-09-18T11:59:07.721292+0000","mapping_epoch":29,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35540992,"num_objects":8677,"num_object_clones":0,"num_object_copies":17354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10808","reported_seq":10887,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.701702+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:07.701702+0000","last_peered":"2024-09-18T11:59:07.701702+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:07.701702+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35288274,"num_objects":8617,"num_object_clones":0,"num_object_copies":17234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10806,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5090},{"shards":"0,2","objects":2433}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10745","reported_seq":10830,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.654238+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:07.654238+0000","last_peered":"2024-09-18T11:59:07.654238+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:07.654238+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35337426,"num_objects":8629,"num_object_clones":0,"num_object_copies":17258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10743,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5070},{"shards":"0,2","objects":2503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10931","reported_seq":11156,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:07.742420+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:07.742420+0000","last_peered":"2024-09-18T11:59:07.742420+0000","last_clean":"2024-09-18T11:59:07.742420+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:07.742420+0000","last_undegraded":"2024-09-18T11:59:07.742420+0000","last_fullsized":"2024-09-18T11:59:07.742420+0000","mapping_epoch":29,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35516416,"num_objects":8671,"num_object_clones":0,"num_object_copies":17342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7248,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10786","reported_seq":17914,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:08.102405+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:08.102405+0000","last_peered":"2024-09-18T11:59:08.102405+0000","last_clean":"2024-09-18T11:59:08.102405+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:08.102405+0000","last_undegraded":"2024-09-18T11:59:08.102405+0000","last_fullsized":"2024-09-18T11:59:08.102405+0000","mapping_epoch":38,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35304658,"num_objects":8621,"num_object_clones":0,"num_object_copies":17242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10784,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10802","reported_seq":10874,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.769749+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:07.769749+0000","last_peered":"2024-09-18T11:59:07.769749+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:07.769749+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35407058,"num_objects":8646,"num_object_clones":0,"num_object_copies":17292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10800,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5114},{"shards":"0,2","objects":2457}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10813","reported_seq":10869,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:08.005405+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:08.005405+0000","last_peered":"2024-09-18T11:59:08.005405+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:08.005405+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35431634,"num_objects":8652,"num_object_clones":0,"num_object_copies":17304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10811,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2471},{"shards":"2","objects":5102}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10831","reported_seq":11040,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.944255+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:59:07.944255+0000","last_peered":"2024-09-18T11:59:07.944255+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:07.944255+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35500032,"num_objects":8667,"num_object_clones":0,"num_object_copies":17334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7311,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2513},{"shards":"2","objects":5074}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10880","reported_seq":10962,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.721229+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:07.721229+0000","last_peered":"2024-09-18T11:59:07.721229+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:07.721229+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35422825,"num_objects":8649,"num_object_clones":0,"num_object_copies":17298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10879,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5114},{"shards":"0,2","objects":2420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10834","reported_seq":17391,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:59:07.787493+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:59:07.787493+0000","last_peered":"2024-09-18T11:59:07.787493+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:59:07.787493+0000","last_undegraded":"2024-09-18T11:59:07.787493+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35587899,"num_objects":8691,"num_object_clones":0,"num_object_copies":17382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":289,"num_objects_unfound":0,"num_objects_dirty":8691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10831,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3239,"num_bytes_recovered":13242368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8402},{"shards":"0,2","objects":289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10683","reported_seq":10764,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.744328+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:07.744328+0000","last_peered":"2024-09-18T11:59:07.744328+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:07.744328+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34878674,"num_objects":8517,"num_object_clones":0,"num_object_copies":17034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10681,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4924},{"shards":"0,2","objects":2511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10854","reported_seq":10939,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.691842+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:07.691842+0000","last_peered":"2024-09-18T11:59:07.691842+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:07.691842+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35465636,"num_objects":8662,"num_object_clones":0,"num_object_copies":17324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10851,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5139},{"shards":"0,2","objects":2430}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":566435916,"num_objects":138308,"num_object_clones":0,"num_object_copies":276616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":289,"num_objects_unfound":0,"num_objects_dirty":138308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156907,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6739,"num_bytes_recovered":27501161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1204535296,"data_stored":1204457572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160714,"ondisk_log_size":160714,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051647,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700968,"kb_used_data":390624,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93670872,"statfs":{"total":96636764160,"available":95918972928,"internally_reserved":0,"allocated":399998976,"data_stored":399771329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.30399999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855410,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621400,"kb_used_data":315216,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93750440,"statfs":{"total":96636764160,"available":96000450560,"internally_reserved":0,"allocated":322781184,"data_stored":322490772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.23400000000000001},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.38300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.26300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769892,"kb_used_data":472088,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93601948,"statfs":{"total":96636764160,"available":95848394752,"internally_reserved":0,"allocated":483418112,"data_stored":483190915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.312},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":482594816,"data_stored":482591337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322433024,"data_stored":322366330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":399507456,"data_stored":399499905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:11.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:12.161 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:12.172 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:12.184 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-18T11:59:11.133405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":566435935,"num_objects":138310,"num_object_clones":0,"num_object_copies":276620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":289,"num_objects_unfound":0,"num_objects_dirty":138310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156909,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6740,"num_bytes_recovered":27501180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160717,"ondisk_log_size":160717,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2092260,"kb_used_data":1177928,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281023260,"statfs":{"total":289910292480,"available":287767818240,"internally_reserved":0,"allocated":1206198272,"data_stored":1205453016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,9,9,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16936960,"num_objects":-4135,"num_object_clones":0,"num_object_copies":-8270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-370,"num_objects_unfound":0,"num_objects_dirty":-4135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4135,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":370,"num_bytes_recovered":1515520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005790"},"pg_stats":[{"pgid":"2.f","version":"39'10807","reported_seq":10878,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.683024+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:07.683024+0000","last_peered":"2024-09-18T11:59:07.683024+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:07.683024+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35753984,"num_objects":8729,"num_object_clones":0,"num_object_copies":17458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5130},{"shards":"0,2","objects":2560}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10799","reported_seq":10955,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:08.060881+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:08.060881+0000","last_peered":"2024-09-18T11:59:08.060881+0000","last_clean":"2024-09-18T11:59:08.060881+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:08.060881+0000","last_undegraded":"2024-09-18T11:59:08.060881+0000","last_fullsized":"2024-09-18T11:59:08.060881+0000","mapping_epoch":29,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35360768,"num_objects":8633,"num_object_clones":0,"num_object_copies":17266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7991,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10860","reported_seq":11046,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:08.079539+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:08.079539+0000","last_peered":"2024-09-18T11:59:08.079539+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:08.079539+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35622936,"num_objects":8698,"num_object_clones":0,"num_object_copies":17396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8698,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7718,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2456},{"shards":"2","objects":5162}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10711","reported_seq":10740,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:08.025504+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:08.025504+0000","last_peered":"2024-09-18T11:59:08.025504+0000","last_clean":"2024-09-18T11:59:08.025504+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:08.025504+0000","last_undegraded":"2024-09-18T11:59:08.025504+0000","last_fullsized":"2024-09-18T11:59:08.025504+0000","mapping_epoch":29,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35016704,"num_objects":8549,"num_object_clones":0,"num_object_copies":17098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10711,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10770","reported_seq":10961,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:07.721292+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:07.721292+0000","last_peered":"2024-09-18T11:59:07.721292+0000","last_clean":"2024-09-18T11:59:07.721292+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:07.721292+0000","last_undegraded":"2024-09-18T11:59:07.721292+0000","last_fullsized":"2024-09-18T11:59:07.721292+0000","mapping_epoch":29,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35540992,"num_objects":8677,"num_object_clones":0,"num_object_copies":17354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10808","reported_seq":10887,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.701702+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:07.701702+0000","last_peered":"2024-09-18T11:59:07.701702+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:07.701702+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35288274,"num_objects":8617,"num_object_clones":0,"num_object_copies":17234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10806,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5090},{"shards":"0,2","objects":2433}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10745","reported_seq":10830,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.654238+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:07.654238+0000","last_peered":"2024-09-18T11:59:07.654238+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:07.654238+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35337426,"num_objects":8629,"num_object_clones":0,"num_object_copies":17258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10743,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5070},{"shards":"0,2","objects":2503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'10931","reported_seq":11156,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:07.742420+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:07.742420+0000","last_peered":"2024-09-18T11:59:07.742420+0000","last_clean":"2024-09-18T11:59:07.742420+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:07.742420+0000","last_undegraded":"2024-09-18T11:59:07.742420+0000","last_fullsized":"2024-09-18T11:59:07.742420+0000","mapping_epoch":29,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35516416,"num_objects":8671,"num_object_clones":0,"num_object_copies":17342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7248,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10786","reported_seq":17914,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:08.102405+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:08.102405+0000","last_peered":"2024-09-18T11:59:08.102405+0000","last_clean":"2024-09-18T11:59:08.102405+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:08.102405+0000","last_undegraded":"2024-09-18T11:59:08.102405+0000","last_fullsized":"2024-09-18T11:59:08.102405+0000","mapping_epoch":38,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35304658,"num_objects":8621,"num_object_clones":0,"num_object_copies":17242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10784,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10802","reported_seq":10874,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.769749+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:07.769749+0000","last_peered":"2024-09-18T11:59:07.769749+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:07.769749+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35407058,"num_objects":8646,"num_object_clones":0,"num_object_copies":17292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10800,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5114},{"shards":"0,2","objects":2457}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10813","reported_seq":10869,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:08.005405+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:08.005405+0000","last_peered":"2024-09-18T11:59:08.005405+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:08.005405+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35431634,"num_objects":8652,"num_object_clones":0,"num_object_copies":17304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10811,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2471},{"shards":"2","objects":5102}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10831","reported_seq":11040,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.944255+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:59:07.944255+0000","last_peered":"2024-09-18T11:59:07.944255+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:07.944255+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35500032,"num_objects":8667,"num_object_clones":0,"num_object_copies":17334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7311,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2513},{"shards":"2","objects":5074}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'10880","reported_seq":10962,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.721229+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:07.721229+0000","last_peered":"2024-09-18T11:59:07.721229+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:07.721229+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35422825,"num_objects":8649,"num_object_clones":0,"num_object_copies":17298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10879,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5114},{"shards":"0,2","objects":2420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10834","reported_seq":17391,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:59:07.787493+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:59:07.787493+0000","last_peered":"2024-09-18T11:59:07.787493+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:59:07.787493+0000","last_undegraded":"2024-09-18T11:59:07.787493+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35587899,"num_objects":8691,"num_object_clones":0,"num_object_copies":17382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":289,"num_objects_unfound":0,"num_objects_dirty":8691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10831,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3239,"num_bytes_recovered":13242368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8402},{"shards":"0,2","objects":289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10683","reported_seq":10764,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.744328+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:07.744328+0000","last_peered":"2024-09-18T11:59:07.744328+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:07.744328+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'600","ondisk_log_start":"30'600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34878674,"num_objects":8517,"num_object_clones":0,"num_object_copies":17034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10681,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4924},{"shards":"0,2","objects":2511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10854","reported_seq":10939,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:07.691842+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:07.691842+0000","last_peered":"2024-09-18T11:59:07.691842+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:07.691842+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35465636,"num_objects":8662,"num_object_clones":0,"num_object_copies":17324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10851,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5139},{"shards":"0,2","objects":2430}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":566435916,"num_objects":138308,"num_object_clones":0,"num_object_copies":276616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":289,"num_objects_unfound":0,"num_objects_dirty":138308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156907,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6739,"num_bytes_recovered":27501161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1204535296,"data_stored":1204457572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160714,"ondisk_log_size":160714,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051647,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700968,"kb_used_data":390624,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93670872,"statfs":{"total":96636764160,"available":95918972928,"internally_reserved":0,"allocated":399998976,"data_stored":399771329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.30399999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.7789999999999999,"15min":1.9810000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.9850000000000003,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.149,"5min":0.98999999999999999,"15min":0.90900000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.7489999999999997,"5min":8.0640000000000001,"15min":8.0640000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855410,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621400,"kb_used_data":315216,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93750440,"statfs":{"total":96636764160,"available":96000450560,"internally_reserved":0,"allocated":322781184,"data_stored":322490772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.23400000000000001},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.38300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.26300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769892,"kb_used_data":472088,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93601948,"statfs":{"total":96636764160,"available":95848394752,"internally_reserved":0,"allocated":483418112,"data_stored":483190915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.312},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":482594816,"data_stored":482591337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322433024,"data_stored":322366330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":399507456,"data_stored":399499905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:12.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:59:12.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'10807", 'reported_seq': 10878, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:07.683024+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:59:07.683024+0000', 'last_peered': '2024-09-18T11:59:07.683024+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:59:07.683024+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'800", 'ondisk_log_start': "30'800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35753984, 'num_objects': 8729, 'num_object_clones': 0, 'num_object_copies': 17458, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10807, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5130}, {'shards': '0,2', 'objects': 2560}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:12.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:59:12.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'10860", 'reported_seq': 11046, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:08.079539+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:59:08.079539+0000', 'last_peered': '2024-09-18T11:59:08.079539+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:59:08.079539+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'800", 'ondisk_log_start': "30'800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35622936, 'num_objects': 8698, 'num_object_clones': 0, 'num_object_copies': 17396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8698, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7718, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2456}, {'shards': '2', 'objects': 5162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:12.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:59:12.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'10808", 'reported_seq': 10887, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:07.701702+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:59:07.701702+0000', 'last_peered': '2024-09-18T11:59:07.701702+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:59:07.701702+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'800", 'ondisk_log_start': "30'800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35288274, 'num_objects': 8617, 'num_object_clones': 0, 'num_object_copies': 17234, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10806, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5090}, {'shards': '0,2', 'objects': 2433}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:12.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:59:12.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'10745", 'reported_seq': 10830, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:07.654238+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:59:07.654238+0000', 'last_peered': '2024-09-18T11:59:07.654238+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:59:07.654238+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'700", 'ondisk_log_start': "30'700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35337426, 'num_objects': 8629, 'num_object_clones': 0, 'num_object_copies': 17258, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10743, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5070}, {'shards': '0,2', 'objects': 2503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:12.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:59:12.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'10802", 'reported_seq': 10874, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:07.769749+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:59:07.769749+0000', 'last_peered': '2024-09-18T11:59:07.769749+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:59:07.769749+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'800", 'ondisk_log_start': "30'800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35407058, 'num_objects': 8646, 'num_object_clones': 0, 'num_object_copies': 17292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10800, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5114}, {'shards': '0,2', 'objects': 2457}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:12.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:59:12.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'10813", 'reported_seq': 10869, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:08.005405+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:59:08.005405+0000', 'last_peered': '2024-09-18T11:59:08.005405+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:59:08.005405+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'800", 'ondisk_log_start': "30'800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35431634, 'num_objects': 8652, 'num_object_clones': 0, 'num_object_copies': 17304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8652, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10811, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2471}, {'shards': '2', 'objects': 5102}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:12.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:59:12.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'10831", 'reported_seq': 11040, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:07.944255+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:59:07.944255+0000', 'last_peered': '2024-09-18T11:59:07.944255+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:59:07.944255+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'800", 'ondisk_log_start': "30'800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35500032, 'num_objects': 8667, 'num_object_clones': 0, 'num_object_copies': 17334, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7311, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2513}, {'shards': '2', 'objects': 5074}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:12.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:59:12.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'10880", 'reported_seq': 10962, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:07.721229+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:59:07.721229+0000', 'last_peered': '2024-09-18T11:59:07.721229+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:59:07.721229+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'800", 'ondisk_log_start': "30'800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35422825, 'num_objects': 8649, 'num_object_clones': 0, 'num_object_copies': 17298, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10879, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5114}, {'shards': '0,2', 'objects': 2420}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:12.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:59:12.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'10834", 'reported_seq': 17391, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:59:07.787493+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:59:07.787493+0000', 'last_peered': '2024-09-18T11:59:07.787493+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:59:07.787493+0000', 'last_undegraded': '2024-09-18T11:59:07.787493+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "30'800", 'ondisk_log_start': "30'800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35587899, 'num_objects': 8691, 'num_object_clones': 0, 'num_object_copies': 17382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 289, 'num_objects_unfound': 0, 'num_objects_dirty': 8691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10831, 'num_write_kb': 39032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3239, 'num_bytes_recovered': 13242368, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 8402}, {'shards': '0,2', 'objects': 289}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:12.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:59:12.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'10683", 'reported_seq': 10764, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:07.744328+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:59:07.744328+0000', 'last_peered': '2024-09-18T11:59:07.744328+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:59:07.744328+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'600", 'ondisk_log_start': "30'600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34878674, 'num_objects': 8517, 'num_object_clones': 0, 'num_object_copies': 17034, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10681, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4924}, {'shards': '0,2', 'objects': 2511}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:12.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:59:12.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'10854", 'reported_seq': 10939, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:07.691842+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:59:07.691842+0000', 'last_peered': '2024-09-18T11:59:07.691842+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:59:07.691842+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'800", 'ondisk_log_start': "30'800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35465636, 'num_objects': 8662, 'num_object_clones': 0, 'num_object_copies': 17324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10851, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5139}, {'shards': '0,2', 'objects': 2430}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:12.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:59:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:13.098 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/276620 objects degraded (12.857%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":353,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":138310,"data_bytes":566435935,"bytes_used":2142474240,"bytes_avail":287767818240,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":276620,"degraded_ratio":0.12856626418914033,"misplaced_objects":289,"misplaced_total":276620,"misplaced_ratio":0.0010447545369098401,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":126232,"recovering_keys_per_sec":0,"num_objects_recovered":370,"num_bytes_recovered":1515520,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":344},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:59:13.135 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:59:13.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:14.004 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:14.018 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:14.020 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-18T11:59:13.133841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":558276703,"num_objects":136318,"num_object_clones":0,"num_object_copies":272636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":123,"num_objects_unfound":0,"num_objects_dirty":136318,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158901,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6906,"num_bytes_recovered":28181116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160909,"ondisk_log_size":160909,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2077400,"kb_used_data":1167232,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281038120,"statfs":{"total":289910292480,"available":287783034880,"internally_reserved":0,"allocated":1195245568,"data_stored":1194500312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4,10,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25096192,"num_objects":-6127,"num_object_clones":0,"num_object_copies":-12254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-536,"num_objects_unfound":0,"num_objects_dirty":-6127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6127,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":536,"num_bytes_recovered":2195456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005731"},"pg_stats":[{"pgid":"2.f","version":"39'10945","reported_seq":11016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.681204+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:12.681204+0000","last_peered":"2024-09-18T11:59:12.681204+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:12.681204+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35188736,"num_objects":8591,"num_object_clones":0,"num_object_copies":17182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4992},{"shards":"0,2","objects":2422}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10943","reported_seq":11101,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:12.980787+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:12.980787+0000","last_peered":"2024-09-18T11:59:12.980787+0000","last_clean":"2024-09-18T11:59:12.980787+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:12.980787+0000","last_undegraded":"2024-09-18T11:59:12.980787+0000","last_fullsized":"2024-09-18T11:59:12.980787+0000","mapping_epoch":29,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34770944,"num_objects":8489,"num_object_clones":0,"num_object_copies":16978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8135,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10969","reported_seq":11155,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:13.094082+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:13.094082+0000","last_peered":"2024-09-18T11:59:13.094082+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:13.094082+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35176472,"num_objects":8589,"num_object_clones":0,"num_object_copies":17178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7827,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2347},{"shards":"2","objects":5053}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10848","reported_seq":10877,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:13.035551+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:13.035551+0000","last_peered":"2024-09-18T11:59:13.035551+0000","last_clean":"2024-09-18T11:59:13.035551+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:13.035551+0000","last_undegraded":"2024-09-18T11:59:13.035551+0000","last_fullsized":"2024-09-18T11:59:13.035551+0000","mapping_epoch":29,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34455552,"num_objects":8412,"num_object_clones":0,"num_object_copies":16824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10848,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10888","reported_seq":11079,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:12.753843+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:12.753843+0000","last_peered":"2024-09-18T11:59:12.753843+0000","last_clean":"2024-09-18T11:59:12.753843+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:12.753843+0000","last_undegraded":"2024-09-18T11:59:12.753843+0000","last_fullsized":"2024-09-18T11:59:12.753843+0000","mapping_epoch":29,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35057664,"num_objects":8559,"num_object_clones":0,"num_object_copies":17118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8053,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10957","reported_seq":11038,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.754160+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:12.754160+0000","last_peered":"2024-09-18T11:59:12.754160+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:12.754160+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34677970,"num_objects":8468,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10955,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4941},{"shards":"0,2","objects":2284}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10870","reported_seq":10956,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.818521+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:12.818521+0000","last_peered":"2024-09-18T11:59:12.818521+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:12.818521+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34825426,"num_objects":8504,"num_object_clones":0,"num_object_copies":17008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10868,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4945},{"shards":"0,2","objects":2378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'11077","reported_seq":11302,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:12.762070+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:12.762070+0000","last_peered":"2024-09-18T11:59:12.762070+0000","last_clean":"2024-09-18T11:59:12.762070+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:12.762070+0000","last_undegraded":"2024-09-18T11:59:12.762070+0000","last_fullsized":"2024-09-18T11:59:12.762070+0000","mapping_epoch":29,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34918400,"num_objects":8525,"num_object_clones":0,"num_object_copies":17050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7394,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10786","reported_seq":17914,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:08.102405+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:08.102405+0000","last_peered":"2024-09-18T11:59:08.102405+0000","last_clean":"2024-09-18T11:59:08.102405+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:08.102405+0000","last_undegraded":"2024-09-18T11:59:08.102405+0000","last_fullsized":"2024-09-18T11:59:08.102405+0000","mapping_epoch":38,"log_start":"30'700","ondisk_log_start":"30'700","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35304658,"num_objects":8621,"num_object_clones":0,"num_object_copies":17242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10784,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10943","reported_seq":11021,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.791233+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:12.791233+0000","last_peered":"2024-09-18T11:59:12.791233+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:12.791233+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34829522,"num_objects":8505,"num_object_clones":0,"num_object_copies":17010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4973},{"shards":"0,2","objects":2316}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10967","reported_seq":11023,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:13.073005+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:13.073005+0000","last_peered":"2024-09-18T11:59:13.073005+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:13.073005+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34800850,"num_objects":8498,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10965,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2317},{"shards":"2","objects":4948}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10950","reported_seq":11159,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:13.006316+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:59:13.006316+0000","last_peered":"2024-09-18T11:59:13.006316+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:13.006316+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35012608,"num_objects":8548,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7430,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2394},{"shards":"2","objects":4955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'11008","reported_seq":11090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.779570+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:12.779570+0000","last_peered":"2024-09-18T11:59:12.779570+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:12.779570+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34898537,"num_objects":8521,"num_object_clones":0,"num_object_copies":17042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11007,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4986},{"shards":"0,2","objects":2292}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10989","reported_seq":17878,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:59:12.803012+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:59:12.803012+0000","last_peered":"2024-09-18T11:59:12.803012+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:59:12.803012+0000","last_undegraded":"2024-09-18T11:59:12.803012+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34953019,"num_objects":8536,"num_object_clones":0,"num_object_copies":17072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":123,"num_objects_unfound":0,"num_objects_dirty":8536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10986,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3405,"num_bytes_recovered":13922304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8413},{"shards":"0,2","objects":123}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10810","reported_seq":10890,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.793398+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:12.793398+0000","last_peered":"2024-09-18T11:59:12.793398+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:12.793398+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34358482,"num_objects":8390,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10808,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4797},{"shards":"0,2","objects":2384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10956","reported_seq":11041,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.770588+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:12.770588+0000","last_peered":"2024-09-18T11:59:12.770588+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:12.770588+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35047844,"num_objects":8560,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10953,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5037},{"shards":"0,2","objects":2328}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":558276684,"num_objects":136316,"num_object_clones":0,"num_object_copies":272632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":123,"num_objects_unfound":0,"num_objects_dirty":136316,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158899,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6905,"num_bytes_recovered":28181097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1193590784,"data_stored":1193513060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051648,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684280,"kb_used_data":386416,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93687560,"statfs":{"total":96636764160,"available":95936061440,"internally_reserved":0,"allocated":395689984,"data_stored":395462337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.20300000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.379}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.23200000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855410,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621400,"kb_used_data":315216,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93750440,"statfs":{"total":96636764160,"available":96000450560,"internally_reserved":0,"allocated":322781184,"data_stored":322490772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.434,"15min":0.46000000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.6579999999999999,"5min":2.7280000000000002,"15min":2.7280000000000002},"last":0.23400000000000001},{"interface":"front","average":{"1min":2.206,"5min":1.165,"15min":1.0409999999999999},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":11.864000000000001,"5min":11.864000000000001,"15min":11.864000000000001},"last":0.38300000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1479999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.133,"5min":7.7629999999999999,"15min":7.7629999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.381,"5min":0.80800000000000005,"15min":0.68700000000000006},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.996,"5min":11.996,"15min":11.996},"last":0.26300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771720,"kb_used_data":465600,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93600120,"statfs":{"total":96636764160,"available":95846522880,"internally_reserved":0,"allocated":476774400,"data_stored":476547203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":5.8840000000000003},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":5.7089999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":6.7720000000000002},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":9.7970000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":476000256,"data_stored":475996777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":322433024,"data_stored":322366330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":395157504,"data_stored":395149953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:17.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:17.751 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:17.756 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:17.768 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-18T11:59:17.138285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":557711455,"num_objects":136180,"num_object_clones":0,"num_object_copies":272360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":123,"num_objects_unfound":0,"num_objects_dirty":136180,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159039,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6906,"num_bytes_recovered":28181116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160847,"ondisk_log_size":160847,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2081128,"kb_used_data":1166800,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281034392,"statfs":{"total":289910292480,"available":287779217408,"internally_reserved":0,"allocated":1194803200,"data_stored":1194057944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,3,11,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8724480,"num_objects":-2130,"num_object_clones":0,"num_object_copies":-4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-166,"num_objects_unfound":0,"num_objects_dirty":-2130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2130,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":679936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005405"},"pg_stats":[{"pgid":"2.f","version":"39'10945","reported_seq":11016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.681204+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:12.681204+0000","last_peered":"2024-09-18T11:59:12.681204+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:12.681204+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35188736,"num_objects":8591,"num_object_clones":0,"num_object_copies":17182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4992},{"shards":"0,2","objects":2422}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10943","reported_seq":11101,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:12.980787+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:12.980787+0000","last_peered":"2024-09-18T11:59:12.980787+0000","last_clean":"2024-09-18T11:59:12.980787+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:12.980787+0000","last_undegraded":"2024-09-18T11:59:12.980787+0000","last_fullsized":"2024-09-18T11:59:12.980787+0000","mapping_epoch":29,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34770944,"num_objects":8489,"num_object_clones":0,"num_object_copies":16978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8135,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10969","reported_seq":11155,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:13.094082+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:13.094082+0000","last_peered":"2024-09-18T11:59:13.094082+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:13.094082+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35176472,"num_objects":8589,"num_object_clones":0,"num_object_copies":17178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7827,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2347},{"shards":"2","objects":5053}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10848","reported_seq":10877,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:13.035551+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:13.035551+0000","last_peered":"2024-09-18T11:59:13.035551+0000","last_clean":"2024-09-18T11:59:13.035551+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:13.035551+0000","last_undegraded":"2024-09-18T11:59:13.035551+0000","last_fullsized":"2024-09-18T11:59:13.035551+0000","mapping_epoch":29,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34455552,"num_objects":8412,"num_object_clones":0,"num_object_copies":16824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10848,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10888","reported_seq":11079,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:12.753843+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:12.753843+0000","last_peered":"2024-09-18T11:59:12.753843+0000","last_clean":"2024-09-18T11:59:12.753843+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:12.753843+0000","last_undegraded":"2024-09-18T11:59:12.753843+0000","last_fullsized":"2024-09-18T11:59:12.753843+0000","mapping_epoch":29,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35057664,"num_objects":8559,"num_object_clones":0,"num_object_copies":17118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8053,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10957","reported_seq":11038,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.754160+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:12.754160+0000","last_peered":"2024-09-18T11:59:12.754160+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:12.754160+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34677970,"num_objects":8468,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10955,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4941},{"shards":"0,2","objects":2284}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10870","reported_seq":10956,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.818521+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:12.818521+0000","last_peered":"2024-09-18T11:59:12.818521+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:12.818521+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34825426,"num_objects":8504,"num_object_clones":0,"num_object_copies":17008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10868,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4945},{"shards":"0,2","objects":2378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'11077","reported_seq":11302,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:12.762070+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:12.762070+0000","last_peered":"2024-09-18T11:59:12.762070+0000","last_clean":"2024-09-18T11:59:12.762070+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:12.762070+0000","last_undegraded":"2024-09-18T11:59:12.762070+0000","last_fullsized":"2024-09-18T11:59:12.762070+0000","mapping_epoch":29,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34918400,"num_objects":8525,"num_object_clones":0,"num_object_copies":17050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7394,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10924","reported_seq":18051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:13.162292+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:13.162292+0000","last_peered":"2024-09-18T11:59:13.162292+0000","last_clean":"2024-09-18T11:59:13.162292+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:13.162292+0000","last_undegraded":"2024-09-18T11:59:13.162292+0000","last_fullsized":"2024-09-18T11:59:13.162292+0000","mapping_epoch":38,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34739410,"num_objects":8483,"num_object_clones":0,"num_object_copies":16966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10922,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10943","reported_seq":11021,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.791233+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:12.791233+0000","last_peered":"2024-09-18T11:59:12.791233+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:12.791233+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34829522,"num_objects":8505,"num_object_clones":0,"num_object_copies":17010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4973},{"shards":"0,2","objects":2316}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10967","reported_seq":11023,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:13.073005+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:13.073005+0000","last_peered":"2024-09-18T11:59:13.073005+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:13.073005+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34800850,"num_objects":8498,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10965,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2317},{"shards":"2","objects":4948}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10950","reported_seq":11159,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:13.006316+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:59:13.006316+0000","last_peered":"2024-09-18T11:59:13.006316+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:13.006316+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35012608,"num_objects":8548,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7430,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2394},{"shards":"2","objects":4955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'11008","reported_seq":11090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.779570+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:12.779570+0000","last_peered":"2024-09-18T11:59:12.779570+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:12.779570+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34898537,"num_objects":8521,"num_object_clones":0,"num_object_copies":17042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11007,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4986},{"shards":"0,2","objects":2292}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10989","reported_seq":17878,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:59:12.803012+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:59:12.803012+0000","last_peered":"2024-09-18T11:59:12.803012+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:59:12.803012+0000","last_undegraded":"2024-09-18T11:59:12.803012+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34953019,"num_objects":8536,"num_object_clones":0,"num_object_copies":17072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":123,"num_objects_unfound":0,"num_objects_dirty":8536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10986,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3405,"num_bytes_recovered":13922304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8413},{"shards":"0,2","objects":123}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10810","reported_seq":10890,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.793398+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:12.793398+0000","last_peered":"2024-09-18T11:59:12.793398+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:12.793398+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34358482,"num_objects":8390,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10808,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4797},{"shards":"0,2","objects":2384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10956","reported_seq":11041,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.770588+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:12.770588+0000","last_peered":"2024-09-18T11:59:12.770588+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:12.770588+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35047844,"num_objects":8560,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10953,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5037},{"shards":"0,2","objects":2328}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":557711436,"num_objects":136178,"num_object_clones":0,"num_object_copies":272356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":123,"num_objects_unfound":0,"num_objects_dirty":136178,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159037,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6905,"num_bytes_recovered":28181097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1193152512,"data_stored":1193074788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160844,"ondisk_log_size":160844,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051648,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684280,"kb_used_data":386416,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93687560,"statfs":{"total":96636764160,"available":95936061440,"internally_reserved":0,"allocated":395689984,"data_stored":395462337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.20300000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.379}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.23200000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855411,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625128,"kb_used_data":314784,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93746712,"statfs":{"total":96636764160,"available":95996633088,"internally_reserved":0,"allocated":322338816,"data_stored":322048404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.214},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.40200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771720,"kb_used_data":465600,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93600120,"statfs":{"total":96636764160,"available":95846522880,"internally_reserved":0,"allocated":476774400,"data_stored":476547203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":5.8840000000000003},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":5.7089999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":6.7720000000000002},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":9.7970000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":476000256,"data_stored":475996777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":321994752,"data_stored":321928058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":395157504,"data_stored":395149953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:17.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:18.586 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:18.600 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:18.617 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-18T11:59:17.138285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":557711455,"num_objects":136180,"num_object_clones":0,"num_object_copies":272360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":123,"num_objects_unfound":0,"num_objects_dirty":136180,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159039,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6906,"num_bytes_recovered":28181116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160847,"ondisk_log_size":160847,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2081128,"kb_used_data":1166800,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281034392,"statfs":{"total":289910292480,"available":287779217408,"internally_reserved":0,"allocated":1194803200,"data_stored":1194057944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,3,11,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8724480,"num_objects":-2130,"num_object_clones":0,"num_object_copies":-4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-166,"num_objects_unfound":0,"num_objects_dirty":-2130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2130,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":679936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005405"},"pg_stats":[{"pgid":"2.f","version":"39'10945","reported_seq":11016,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.681204+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:12.681204+0000","last_peered":"2024-09-18T11:59:12.681204+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:12.681204+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35188736,"num_objects":8591,"num_object_clones":0,"num_object_copies":17182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4992},{"shards":"0,2","objects":2422}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"39'10943","reported_seq":11101,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:12.980787+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:12.980787+0000","last_peered":"2024-09-18T11:59:12.980787+0000","last_clean":"2024-09-18T11:59:12.980787+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:12.980787+0000","last_undegraded":"2024-09-18T11:59:12.980787+0000","last_fullsized":"2024-09-18T11:59:12.980787+0000","mapping_epoch":29,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34770944,"num_objects":8489,"num_object_clones":0,"num_object_copies":16978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8135,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'10969","reported_seq":11155,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:13.094082+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:13.094082+0000","last_peered":"2024-09-18T11:59:13.094082+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:13.094082+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35176472,"num_objects":8589,"num_object_clones":0,"num_object_copies":17178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8589,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7827,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2347},{"shards":"2","objects":5053}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"39'10848","reported_seq":10877,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:13.035551+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:13.035551+0000","last_peered":"2024-09-18T11:59:13.035551+0000","last_clean":"2024-09-18T11:59:13.035551+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:13.035551+0000","last_undegraded":"2024-09-18T11:59:13.035551+0000","last_fullsized":"2024-09-18T11:59:13.035551+0000","mapping_epoch":29,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34455552,"num_objects":8412,"num_object_clones":0,"num_object_copies":16824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10848,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'10888","reported_seq":11079,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:12.753843+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:12.753843+0000","last_peered":"2024-09-18T11:59:12.753843+0000","last_clean":"2024-09-18T11:59:12.753843+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:12.753843+0000","last_undegraded":"2024-09-18T11:59:12.753843+0000","last_fullsized":"2024-09-18T11:59:12.753843+0000","mapping_epoch":29,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35057664,"num_objects":8559,"num_object_clones":0,"num_object_copies":17118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8053,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'10957","reported_seq":11038,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.754160+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:12.754160+0000","last_peered":"2024-09-18T11:59:12.754160+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:12.754160+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34677970,"num_objects":8468,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10955,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4941},{"shards":"0,2","objects":2284}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'10870","reported_seq":10956,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.818521+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:12.818521+0000","last_peered":"2024-09-18T11:59:12.818521+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:12.818521+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34825426,"num_objects":8504,"num_object_clones":0,"num_object_copies":17008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10868,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4945},{"shards":"0,2","objects":2378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'11077","reported_seq":11302,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:12.762070+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:12.762070+0000","last_peered":"2024-09-18T11:59:12.762070+0000","last_clean":"2024-09-18T11:59:12.762070+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:12.762070+0000","last_undegraded":"2024-09-18T11:59:12.762070+0000","last_fullsized":"2024-09-18T11:59:12.762070+0000","mapping_epoch":29,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34918400,"num_objects":8525,"num_object_clones":0,"num_object_copies":17050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7394,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470086+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:56:56.470086+0000","last_peered":"2024-09-18T11:56:56.470086+0000","last_clean":"2024-09-18T11:56:56.470086+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:56:56.470086+0000","last_undegraded":"2024-09-18T11:56:56.470086+0000","last_fullsized":"2024-09-18T11:56:56.470086+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'10924","reported_seq":18051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T11:59:13.162292+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:13.162292+0000","last_peered":"2024-09-18T11:59:13.162292+0000","last_clean":"2024-09-18T11:59:13.162292+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:13.162292+0000","last_undegraded":"2024-09-18T11:59:13.162292+0000","last_fullsized":"2024-09-18T11:59:13.162292+0000","mapping_epoch":38,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34739410,"num_objects":8483,"num_object_clones":0,"num_object_copies":16966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10922,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470843+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:56:56.470843+0000","last_peered":"2024-09-18T11:56:56.470843+0000","last_clean":"2024-09-18T11:56:56.470843+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:56:56.470843+0000","last_undegraded":"2024-09-18T11:56:56.470843+0000","last_fullsized":"2024-09-18T11:56:56.470843+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'10943","reported_seq":11021,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.791233+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:12.791233+0000","last_peered":"2024-09-18T11:59:12.791233+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:12.791233+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34829522,"num_objects":8505,"num_object_clones":0,"num_object_copies":17010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4973},{"shards":"0,2","objects":2316}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":91,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.470351+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:56:56.470351+0000","last_peered":"2024-09-18T11:56:56.470351+0000","last_clean":"2024-09-18T11:56:56.470351+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:56:56.470351+0000","last_undegraded":"2024-09-18T11:56:56.470351+0000","last_fullsized":"2024-09-18T11:56:56.470351+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'10967","reported_seq":11023,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:13.073005+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:13.073005+0000","last_peered":"2024-09-18T11:59:13.073005+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:13.073005+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34800850,"num_objects":8498,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10965,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2317},{"shards":"2","objects":4948}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":71,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468402+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:56:56.468402+0000","last_peered":"2024-09-18T11:56:56.468402+0000","last_clean":"2024-09-18T11:56:56.468402+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:56:56.468402+0000","last_undegraded":"2024-09-18T11:56:56.468402+0000","last_fullsized":"2024-09-18T11:56:56.468402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'10950","reported_seq":11159,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:13.006316+0000","last_change":"2024-09-18T11:55:04.672264+0000","last_active":"2024-09-18T11:59:13.006316+0000","last_peered":"2024-09-18T11:59:13.006316+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:13.006316+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35012608,"num_objects":8548,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7430,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2394},{"shards":"2","objects":4955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'11008","reported_seq":11090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.779570+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:12.779570+0000","last_peered":"2024-09-18T11:59:12.779570+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:12.779570+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34898537,"num_objects":8521,"num_object_clones":0,"num_object_copies":17042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11007,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4986},{"shards":"0,2","objects":2292}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468487+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:56:56.468487+0000","last_peered":"2024-09-18T11:56:56.468487+0000","last_clean":"2024-09-18T11:56:56.468487+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:56:56.468487+0000","last_undegraded":"2024-09-18T11:56:56.468487+0000","last_fullsized":"2024-09-18T11:56:56.468487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'10989","reported_seq":17878,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T11:59:12.803012+0000","last_change":"2024-09-18T11:56:56.323282+0000","last_active":"2024-09-18T11:59:12.803012+0000","last_peered":"2024-09-18T11:59:12.803012+0000","last_clean":"2024-09-18T11:55:02.230955+0000","last_became_active":"2024-09-18T11:55:04.681003+0000","last_became_peered":"2024-09-18T11:55:04.681003+0000","last_unstale":"2024-09-18T11:59:12.803012+0000","last_undegraded":"2024-09-18T11:59:12.803012+0000","last_fullsized":"2024-09-18T11:55:04.248667+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34953019,"num_objects":8536,"num_object_clones":0,"num_object_copies":17072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":123,"num_objects_unfound":0,"num_objects_dirty":8536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10986,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3405,"num_bytes_recovered":13922304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":8413},{"shards":"0,2","objects":123}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.471735+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:56:56.471735+0000","last_peered":"2024-09-18T11:56:56.471735+0000","last_clean":"2024-09-18T11:56:56.471735+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:56:56.471735+0000","last_undegraded":"2024-09-18T11:56:56.471735+0000","last_fullsized":"2024-09-18T11:56:56.471735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'10810","reported_seq":10890,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.793398+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:12.793398+0000","last_peered":"2024-09-18T11:59:12.793398+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:12.793398+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'800","ondisk_log_start":"30'800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34358482,"num_objects":8390,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10808,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4797},{"shards":"0,2","objects":2384}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468492+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:56:56.468492+0000","last_peered":"2024-09-18T11:56:56.468492+0000","last_clean":"2024-09-18T11:56:56.468492+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:56:56.468492+0000","last_undegraded":"2024-09-18T11:56:56.468492+0000","last_fullsized":"2024-09-18T11:56:56.468492+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T11:56:56.468496+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:56:56.468496+0000","last_peered":"2024-09-18T11:56:56.468496+0000","last_clean":"2024-09-18T11:56:56.468496+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:56:56.468496+0000","last_undegraded":"2024-09-18T11:56:56.468496+0000","last_fullsized":"2024-09-18T11:56:56.468496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'10956","reported_seq":11041,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:12.770588+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:12.770588+0000","last_peered":"2024-09-18T11:59:12.770588+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:12.770588+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35047844,"num_objects":8560,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10953,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5037},{"shards":"0,2","objects":2328}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":557711436,"num_objects":136178,"num_object_clones":0,"num_object_copies":272356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35564,"num_objects_misplaced":123,"num_objects_unfound":0,"num_objects_dirty":136178,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159037,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6905,"num_bytes_recovered":28181097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1193152512,"data_stored":1193074788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160844,"ondisk_log_size":160844,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051648,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684280,"kb_used_data":386416,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93687560,"statfs":{"total":96636764160,"available":95936061440,"internally_reserved":0,"allocated":395689984,"data_stored":395462337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.20300000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.379}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.23200000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855411,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625128,"kb_used_data":314784,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93746712,"statfs":{"total":96636764160,"available":95996633088,"internally_reserved":0,"allocated":322338816,"data_stored":322048404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.214},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.40200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771720,"kb_used_data":465600,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93600120,"statfs":{"total":96636764160,"available":95846522880,"internally_reserved":0,"allocated":476774400,"data_stored":476547203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:58:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":0.93799999999999994,"15min":0.999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.3200000000000003,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":5.8840000000000003},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0569999999999999,"15min":1.1259999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":4.617,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":5.7089999999999996}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":6.7720000000000002},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":9.7970000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":476000256,"data_stored":475996777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":321994752,"data_stored":321928058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":395157504,"data_stored":395149953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:18.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:59:18.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'10945", 'reported_seq': 11016, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:12.681204+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:59:12.681204+0000', 'last_peered': '2024-09-18T11:59:12.681204+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:59:12.681204+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'900", 'ondisk_log_start': "30'900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35188736, 'num_objects': 8591, 'num_object_clones': 0, 'num_object_copies': 17182, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10945, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4992}, {'shards': '0,2', 'objects': 2422}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:18.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:59:18.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'10969", 'reported_seq': 11155, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:13.094082+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:59:13.094082+0000', 'last_peered': '2024-09-18T11:59:13.094082+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:59:13.094082+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'900", 'ondisk_log_start': "30'900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35176472, 'num_objects': 8589, 'num_object_clones': 0, 'num_object_copies': 17178, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8589, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7827, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2347}, {'shards': '2', 'objects': 5053}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:18.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:59:18.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'10957", 'reported_seq': 11038, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:12.754160+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:59:12.754160+0000', 'last_peered': '2024-09-18T11:59:12.754160+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:59:12.754160+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'900", 'ondisk_log_start': "30'900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34677970, 'num_objects': 8468, 'num_object_clones': 0, 'num_object_copies': 16936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10955, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4941}, {'shards': '0,2', 'objects': 2284}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:18.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:59:18.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'10870", 'reported_seq': 10956, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:12.818521+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:59:12.818521+0000', 'last_peered': '2024-09-18T11:59:12.818521+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:59:12.818521+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'800", 'ondisk_log_start': "30'800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34825426, 'num_objects': 8504, 'num_object_clones': 0, 'num_object_copies': 17008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8504, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10868, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4945}, {'shards': '0,2', 'objects': 2378}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:18.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:59:18.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'10943", 'reported_seq': 11021, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:12.791233+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:59:12.791233+0000', 'last_peered': '2024-09-18T11:59:12.791233+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:59:12.791233+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'900", 'ondisk_log_start': "30'900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34829522, 'num_objects': 8505, 'num_object_clones': 0, 'num_object_copies': 17010, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8505, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10941, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4973}, {'shards': '0,2', 'objects': 2316}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:18.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:59:18.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'10967", 'reported_seq': 11023, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:13.073005+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:59:13.073005+0000', 'last_peered': '2024-09-18T11:59:13.073005+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:59:13.073005+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'900", 'ondisk_log_start': "30'900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34800850, 'num_objects': 8498, 'num_object_clones': 0, 'num_object_copies': 16996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10965, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2317}, {'shards': '2', 'objects': 4948}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:18.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:59:18.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'10950", 'reported_seq': 11159, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:13.006316+0000', 'last_change': '2024-09-18T11:55:04.672264+0000', 'last_active': '2024-09-18T11:59:13.006316+0000', 'last_peered': '2024-09-18T11:59:13.006316+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:59:13.006316+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'900", 'ondisk_log_start': "30'900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35012608, 'num_objects': 8548, 'num_object_clones': 0, 'num_object_copies': 17096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7430, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2394}, {'shards': '2', 'objects': 4955}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:18.621 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:59:18.621 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'11008", 'reported_seq': 11090, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:12.779570+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:59:12.779570+0000', 'last_peered': '2024-09-18T11:59:12.779570+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:59:12.779570+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'1000", 'ondisk_log_start': "30'1000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34898537, 'num_objects': 8521, 'num_object_clones': 0, 'num_object_copies': 17042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11007, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4986}, {'shards': '0,2', 'objects': 2292}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:18.621 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-18T11:59:18.621 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'10989", 'reported_seq': 17878, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T11:59:12.803012+0000', 'last_change': '2024-09-18T11:56:56.323282+0000', 'last_active': '2024-09-18T11:59:12.803012+0000', 'last_peered': '2024-09-18T11:59:12.803012+0000', 'last_clean': '2024-09-18T11:55:02.230955+0000', 'last_became_active': '2024-09-18T11:55:04.681003+0000', 'last_became_peered': '2024-09-18T11:55:04.681003+0000', 'last_unstale': '2024-09-18T11:59:12.803012+0000', 'last_undegraded': '2024-09-18T11:59:12.803012+0000', 'last_fullsized': '2024-09-18T11:55:04.248667+0000', 'mapping_epoch': 36, 'log_start': "30'900", 'ondisk_log_start': "30'900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34953019, 'num_objects': 8536, 'num_object_clones': 0, 'num_object_copies': 17072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 123, 'num_objects_unfound': 0, 'num_objects_dirty': 8536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10986, 'num_write_kb': 39032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3405, 'num_bytes_recovered': 13922304, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 8413}, {'shards': '0,2', 'objects': 123}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:18.621 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:59:18.621 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'10810", 'reported_seq': 10890, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:12.793398+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:59:12.793398+0000', 'last_peered': '2024-09-18T11:59:12.793398+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:59:12.793398+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'800", 'ondisk_log_start': "30'800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34358482, 'num_objects': 8390, 'num_object_clones': 0, 'num_object_copies': 16780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10808, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4797}, {'shards': '0,2', 'objects': 2384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:18.621 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:59:18.621 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'10956", 'reported_seq': 11041, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:12.770588+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:59:12.770588+0000', 'last_peered': '2024-09-18T11:59:12.770588+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:59:12.770588+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'900", 'ondisk_log_start': "30'900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35047844, 'num_objects': 8560, 'num_object_clones': 0, 'num_object_copies': 17120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10953, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5037}, {'shards': '0,2', 'objects': 2328}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:18.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:59:19.370 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:19.371 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35564/272360 objects degraded (13.058%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":359,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":136180,"data_bytes":557711455,"bytes_used":2131075072,"bytes_avail":287779217408,"bytes_total":289910292480,"degraded_objects":35564,"degraded_total":272360,"degraded_ratio":0.13057717726538404,"misplaced_objects":123,"misplaced_total":272360,"misplaced_ratio":0.00045160816566309299,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":84934,"recovering_keys_per_sec":0,"num_objects_recovered":166,"num_bytes_recovered":679936,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":266},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:59:19.401 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:59:19.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:20.029 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:20.037 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:20.053 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-18T11:59:19.138748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":549425247,"num_objects":134157,"num_object_clones":0,"num_object_copies":268314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33246,"num_objects_misplaced":2256,"num_objects_unfound":0,"num_objects_dirty":134157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161062,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7089,"num_bytes_recovered":28930684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160970,"ondisk_log_size":160970,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2070280,"kb_used_data":1155928,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281045240,"statfs":{"total":289910292480,"available":287790325760,"internally_reserved":0,"allocated":1183670272,"data_stored":1182893609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,9,6,4,5,5,2,3,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17010688,"num_objects":-4153,"num_object_clones":0,"num_object_copies":-8306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2318,"num_objects_misplaced":1967,"num_objects_unfound":0,"num_objects_dirty":-4153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4153,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":349,"num_bytes_recovered":1429504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005342"},"pg_stats":[{"pgid":"2.f","version":"41'11057","reported_seq":11130,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:17.742293+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:17.742293+0000","last_peered":"2024-09-18T11:59:17.742293+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:17.742293+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34729984,"num_objects":8479,"num_object_clones":0,"num_object_copies":16958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11057,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4880},{"shards":"0,2","objects":2310}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11071","reported_seq":11231,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:18.073214+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:18.073214+0000","last_peered":"2024-09-18T11:59:18.073214+0000","last_clean":"2024-09-18T11:59:18.073214+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:18.073214+0000","last_undegraded":"2024-09-18T11:59:18.073214+0000","last_fullsized":"2024-09-18T11:59:18.073214+0000","mapping_epoch":29,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34246656,"num_objects":8361,"num_object_clones":0,"num_object_copies":16722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8263,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11091","reported_seq":11277,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:18.102410+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:18.102410+0000","last_peered":"2024-09-18T11:59:18.102410+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:18.102410+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34676760,"num_objects":8467,"num_object_clones":0,"num_object_copies":16934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8467,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7949,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2225},{"shards":"2","objects":4931}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'10965","reported_seq":10996,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:18.027987+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:18.027987+0000","last_peered":"2024-09-18T11:59:18.027987+0000","last_clean":"2024-09-18T11:59:18.027987+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:18.027987+0000","last_undegraded":"2024-09-18T11:59:18.027987+0000","last_fullsized":"2024-09-18T11:59:18.027987+0000","mapping_epoch":29,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33976320,"num_objects":8295,"num_object_clones":0,"num_object_copies":16590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10965,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11024","reported_seq":11217,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:17.755325+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:17.755325+0000","last_peered":"2024-09-18T11:59:17.755325+0000","last_clean":"2024-09-18T11:59:17.755325+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:17.755325+0000","last_undegraded":"2024-09-18T11:59:17.755325+0000","last_fullsized":"2024-09-18T11:59:17.755325+0000","mapping_epoch":29,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34500608,"num_objects":8423,"num_object_clones":0,"num_object_copies":16846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8189,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11080","reported_seq":11161,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:17.802799+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:17.802799+0000","last_peered":"2024-09-18T11:59:17.802799+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:17.802799+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34174162,"num_objects":8345,"num_object_clones":0,"num_object_copies":16690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11078,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4818},{"shards":"0,2","objects":2161}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'10990","reported_seq":11079,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:17.783241+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:17.783241+0000","last_peered":"2024-09-18T11:59:17.783241+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:17.783241+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34333906,"num_objects":8384,"num_object_clones":0,"num_object_copies":16768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4825},{"shards":"0,2","objects":2258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11198","reported_seq":11425,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:17.750149+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:17.750149+0000","last_peered":"2024-09-18T11:59:17.750149+0000","last_clean":"2024-09-18T11:59:17.750149+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:17.750149+0000","last_undegraded":"2024-09-18T11:59:17.750149+0000","last_fullsized":"2024-09-18T11:59:17.750149+0000","mapping_epoch":29,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34422784,"num_objects":8404,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7515,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11054","reported_seq":18184,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:18.243367+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:18.243367+0000","last_peered":"2024-09-18T11:59:18.243367+0000","last_clean":"2024-09-18T11:59:18.243367+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:18.243367+0000","last_undegraded":"2024-09-18T11:59:18.243367+0000","last_fullsized":"2024-09-18T11:59:18.243367+0000","mapping_epoch":38,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34206930,"num_objects":8353,"num_object_clones":0,"num_object_copies":16706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11052,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11071","reported_seq":11150,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:17.802691+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:17.802691+0000","last_peered":"2024-09-18T11:59:17.802691+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:17.802691+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34305234,"num_objects":8377,"num_object_clones":0,"num_object_copies":16754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11069,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4845},{"shards":"0,2","objects":2188}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11113","reported_seq":11171,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:17.993575+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:17.993575+0000","last_peered":"2024-09-18T11:59:17.993575+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:17.993575+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34202834,"num_objects":8352,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11111,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2171},{"shards":"2","objects":4802}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11087","reported_seq":11418,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:18.088130+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:18.088130+0000","last_peered":"2024-09-18T11:59:18.088130+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:18.088130+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34451456,"num_objects":8411,"num_object_clones":0,"num_object_copies":16822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1277,"num_objects_misplaced":2256,"num_objects_unfound":0,"num_objects_dirty":8411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7567,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":245760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2256},{"shards":"2","objects":4878}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11132","reported_seq":11216,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:17.702413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:17.702413+0000","last_peered":"2024-09-18T11:59:17.702413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:17.702413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34390633,"num_objects":8397,"num_object_clones":0,"num_object_copies":16794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11131,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4862},{"shards":"0,2","objects":2168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11127","reported_seq":18275,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:18.249405+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:18.249405+0000","last_peered":"2024-09-18T11:59:18.249405+0000","last_clean":"2024-09-18T11:59:18.249405+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:18.249405+0000","last_undegraded":"2024-09-18T11:59:18.249405+0000","last_fullsized":"2024-09-18T11:59:18.249405+0000","mapping_epoch":40,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34387771,"num_objects":8398,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11124,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'10927","reported_seq":11010,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:17.596563+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:17.596563+0000","last_peered":"2024-09-18T11:59:17.596563+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:17.596563+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'900","ondisk_log_start":"30'900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33879250,"num_objects":8273,"num_object_clones":0,"num_object_copies":16546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10925,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4680},{"shards":"0,2","objects":2267}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11080","reported_seq":11167,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:17.687393+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:17.687393+0000","last_peered":"2024-09-18T11:59:17.687393+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:17.687393+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34539940,"num_objects":8436,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11077,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4913},{"shards":"0,2","objects":2204}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":549425228,"num_objects":134155,"num_object_clones":0,"num_object_copies":268310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33246,"num_objects_misplaced":2256,"num_objects_unfound":0,"num_objects_dirty":134155,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161060,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7088,"num_bytes_recovered":28930665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1182167040,"data_stored":1182089316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160967,"ondisk_log_size":160967,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051649,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684240,"kb_used_data":382204,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93687600,"statfs":{"total":96636764160,"available":95936102400,"internally_reserved":0,"allocated":391376896,"data_stored":391138780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":2.3199999999999998},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":2.7650000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":2.5390000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":2.355}]}]},{"osd":1,"up_from":35,"seq":150323855412,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616240,"kb_used_data":314212,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93755600,"statfs":{"total":96636764160,"available":96005734400,"internally_reserved":0,"allocated":321753088,"data_stored":321452207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,1,1,2,2,3,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.255},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.28799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":4.819},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":4.7560000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":769800,"kb_used_data":459512,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93602040,"statfs":{"total":96636764160,"available":95848488960,"internally_reserved":0,"allocated":470540288,"data_stored":470302622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,4,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.253}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":469921792,"data_stored":469918313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":321368064,"data_stored":321301370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":390877184,"data_stored":390869633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:23.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:23.791 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:23.794 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:23.831 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-18T11:59:23.139739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":541868127,"num_objects":132312,"num_object_clones":0,"num_object_copies":264624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33289,"num_objects_misplaced":2123,"num_objects_unfound":0,"num_objects_dirty":132312,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162907,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7179,"num_bytes_recovered":29299324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160615,"ondisk_log_size":160615,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2051728,"kb_used_data":1145716,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281063792,"statfs":{"total":289910292480,"available":287809323008,"internally_reserved":0,"allocated":1173213184,"data_stored":1172436521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,4,2,5,10,4,2,3,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15843328,"num_objects":-3868,"num_object_clones":0,"num_object_copies":-7736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2275,"num_objects_misplaced":2000,"num_objects_unfound":0,"num_objects_dirty":-3868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3868,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":1118208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005406"},"pg_stats":[{"pgid":"2.f","version":"41'11210","reported_seq":11283,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.842872+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:22.842872+0000","last_peered":"2024-09-18T11:59:22.842872+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:22.842872+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34103296,"num_objects":8326,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11210,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4727},{"shards":"0,2","objects":2157}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11192","reported_seq":11352,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:23.102077+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:23.102077+0000","last_peered":"2024-09-18T11:59:23.102077+0000","last_clean":"2024-09-18T11:59:23.102077+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:23.102077+0000","last_undegraded":"2024-09-18T11:59:23.102077+0000","last_fullsized":"2024-09-18T11:59:23.102077+0000","mapping_epoch":29,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":8240,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8384,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11205","reported_seq":11393,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:23.063356+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:23.063356+0000","last_peered":"2024-09-18T11:59:23.063356+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:23.063356+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209816,"num_objects":8353,"num_object_clones":0,"num_object_copies":16706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8353,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8063,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2111},{"shards":"2","objects":4817}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11112","reported_seq":11143,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:22.956624+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:22.956624+0000","last_peered":"2024-09-18T11:59:22.956624+0000","last_clean":"2024-09-18T11:59:22.956624+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:22.956624+0000","last_undegraded":"2024-09-18T11:59:22.956624+0000","last_fullsized":"2024-09-18T11:59:22.956624+0000","mapping_epoch":29,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33374208,"num_objects":8148,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11112,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11155","reported_seq":11348,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:22.780969+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:22.780969+0000","last_peered":"2024-09-18T11:59:22.780969+0000","last_clean":"2024-09-18T11:59:22.780969+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:22.780969+0000","last_undegraded":"2024-09-18T11:59:22.780969+0000","last_fullsized":"2024-09-18T11:59:22.780969+0000","mapping_epoch":29,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33964032,"num_objects":8292,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8320,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11213","reported_seq":11295,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.810585+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:22.810585+0000","last_peered":"2024-09-18T11:59:22.810585+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:22.810585+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33629394,"num_objects":8212,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11211,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4685},{"shards":"0,2","objects":2028}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11117","reported_seq":11206,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.652644+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:22.652644+0000","last_peered":"2024-09-18T11:59:22.652644+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:22.652644+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33813714,"num_objects":8257,"num_object_clones":0,"num_object_copies":16514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11115,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4698},{"shards":"0,2","objects":2131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11328","reported_seq":11555,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:22.775071+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:22.775071+0000","last_peered":"2024-09-18T11:59:22.775071+0000","last_clean":"2024-09-18T11:59:22.775071+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:22.775071+0000","last_undegraded":"2024-09-18T11:59:22.775071+0000","last_fullsized":"2024-09-18T11:59:22.775071+0000","mapping_epoch":29,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33890304,"num_objects":8274,"num_object_clones":0,"num_object_copies":16548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11054","reported_seq":18184,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:18.243367+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:18.243367+0000","last_peered":"2024-09-18T11:59:18.243367+0000","last_clean":"2024-09-18T11:59:18.243367+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:18.243367+0000","last_undegraded":"2024-09-18T11:59:18.243367+0000","last_fullsized":"2024-09-18T11:59:18.243367+0000","mapping_epoch":38,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34206930,"num_objects":8353,"num_object_clones":0,"num_object_copies":16706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11052,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11185","reported_seq":11262,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.802050+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:22.802050+0000","last_peered":"2024-09-18T11:59:22.802050+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:22.802050+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33838290,"num_objects":8263,"num_object_clones":0,"num_object_copies":16526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11183,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4731},{"shards":"0,2","objects":2074}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11249","reported_seq":11307,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:23.072997+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:23.072997+0000","last_peered":"2024-09-18T11:59:23.072997+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:23.072997+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33645778,"num_objects":8216,"num_object_clones":0,"num_object_copies":16432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11247,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2035},{"shards":"2","objects":4666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11220","reported_seq":11732,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:23.096973+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:23.096973+0000","last_peered":"2024-09-18T11:59:23.096973+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:23.096973+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33906688,"num_objects":8278,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1320,"num_objects_misplaced":2123,"num_objects_unfound":0,"num_objects_dirty":8278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7700,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2123},{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11251","reported_seq":11335,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.844217+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:22.844217+0000","last_peered":"2024-09-18T11:59:22.844217+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:22.844217+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33903209,"num_objects":8278,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11250,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4743},{"shards":"0,2","objects":2049}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11127","reported_seq":18275,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:18.249405+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:18.249405+0000","last_peered":"2024-09-18T11:59:18.249405+0000","last_clean":"2024-09-18T11:59:18.249405+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:18.249405+0000","last_undegraded":"2024-09-18T11:59:18.249405+0000","last_fullsized":"2024-09-18T11:59:18.249405+0000","mapping_epoch":40,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34387771,"num_objects":8398,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11124,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11076","reported_seq":11159,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.792958+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:22.792958+0000","last_peered":"2024-09-18T11:59:22.792958+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:22.792958+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33268946,"num_objects":8124,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11074,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4531},{"shards":"0,2","objects":2118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11218","reported_seq":11305,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.816942+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:22.816942+0000","last_peered":"2024-09-18T11:59:22.816942+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:22.816942+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33974692,"num_objects":8298,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11215,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4775},{"shards":"0,2","objects":2066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":541868108,"num_objects":132310,"num_object_clones":0,"num_object_copies":264620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33289,"num_objects_misplaced":2123,"num_objects_unfound":0,"num_objects_dirty":132310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162905,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7178,"num_bytes_recovered":29299305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171562496,"data_stored":1171484772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160612,"ondisk_log_size":160612,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051650,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684304,"kb_used_data":378120,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93687536,"statfs":{"total":96636764160,"available":95936036864,"internally_reserved":0,"allocated":387194880,"data_stored":386956764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.189},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.28100000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.252},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.223}]}]},{"osd":1,"up_from":35,"seq":150323855412,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616240,"kb_used_data":314212,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93755600,"statfs":{"total":96636764160,"available":96005734400,"internally_reserved":0,"allocated":321753088,"data_stored":321452207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,1,1,2,2,3,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.255},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.28799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":4.819},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":4.7560000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751184,"kb_used_data":453384,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93620656,"statfs":{"total":96636764160,"available":95867551744,"internally_reserved":0,"allocated":464265216,"data_stored":464027550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,0,2,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.30099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":463437824,"data_stored":463434345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":321368064,"data_stored":321301370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386756608,"data_stored":386749057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:23.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:24.533 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:24.567 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:24.592 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-18T11:59:23.139739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":541868127,"num_objects":132312,"num_object_clones":0,"num_object_copies":264624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33289,"num_objects_misplaced":2123,"num_objects_unfound":0,"num_objects_dirty":132312,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162907,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7179,"num_bytes_recovered":29299324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160615,"ondisk_log_size":160615,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2051728,"kb_used_data":1145716,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281063792,"statfs":{"total":289910292480,"available":287809323008,"internally_reserved":0,"allocated":1173213184,"data_stored":1172436521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,4,2,5,10,4,2,3,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15843328,"num_objects":-3868,"num_object_clones":0,"num_object_copies":-7736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2275,"num_objects_misplaced":2000,"num_objects_unfound":0,"num_objects_dirty":-3868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3868,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":1118208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005406"},"pg_stats":[{"pgid":"2.f","version":"41'11210","reported_seq":11283,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.842872+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:22.842872+0000","last_peered":"2024-09-18T11:59:22.842872+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:22.842872+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34103296,"num_objects":8326,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11210,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4727},{"shards":"0,2","objects":2157}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11192","reported_seq":11352,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:23.102077+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:23.102077+0000","last_peered":"2024-09-18T11:59:23.102077+0000","last_clean":"2024-09-18T11:59:23.102077+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:23.102077+0000","last_undegraded":"2024-09-18T11:59:23.102077+0000","last_fullsized":"2024-09-18T11:59:23.102077+0000","mapping_epoch":29,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":8240,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8384,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11205","reported_seq":11393,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:23.063356+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:23.063356+0000","last_peered":"2024-09-18T11:59:23.063356+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:23.063356+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209816,"num_objects":8353,"num_object_clones":0,"num_object_copies":16706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8353,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8063,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2111},{"shards":"2","objects":4817}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11112","reported_seq":11143,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:22.956624+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:22.956624+0000","last_peered":"2024-09-18T11:59:22.956624+0000","last_clean":"2024-09-18T11:59:22.956624+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:22.956624+0000","last_undegraded":"2024-09-18T11:59:22.956624+0000","last_fullsized":"2024-09-18T11:59:22.956624+0000","mapping_epoch":29,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33374208,"num_objects":8148,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11112,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11155","reported_seq":11348,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:22.780969+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:22.780969+0000","last_peered":"2024-09-18T11:59:22.780969+0000","last_clean":"2024-09-18T11:59:22.780969+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:22.780969+0000","last_undegraded":"2024-09-18T11:59:22.780969+0000","last_fullsized":"2024-09-18T11:59:22.780969+0000","mapping_epoch":29,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33964032,"num_objects":8292,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8320,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11213","reported_seq":11295,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.810585+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:22.810585+0000","last_peered":"2024-09-18T11:59:22.810585+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:22.810585+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33629394,"num_objects":8212,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11211,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4685},{"shards":"0,2","objects":2028}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11117","reported_seq":11206,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.652644+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:22.652644+0000","last_peered":"2024-09-18T11:59:22.652644+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:22.652644+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33813714,"num_objects":8257,"num_object_clones":0,"num_object_copies":16514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11115,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4698},{"shards":"0,2","objects":2131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11328","reported_seq":11555,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:22.775071+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:22.775071+0000","last_peered":"2024-09-18T11:59:22.775071+0000","last_clean":"2024-09-18T11:59:22.775071+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:22.775071+0000","last_undegraded":"2024-09-18T11:59:22.775071+0000","last_fullsized":"2024-09-18T11:59:22.775071+0000","mapping_epoch":29,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33890304,"num_objects":8274,"num_object_clones":0,"num_object_copies":16548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11054","reported_seq":18184,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:18.243367+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:18.243367+0000","last_peered":"2024-09-18T11:59:18.243367+0000","last_clean":"2024-09-18T11:59:18.243367+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:18.243367+0000","last_undegraded":"2024-09-18T11:59:18.243367+0000","last_fullsized":"2024-09-18T11:59:18.243367+0000","mapping_epoch":38,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34206930,"num_objects":8353,"num_object_clones":0,"num_object_copies":16706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11052,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11185","reported_seq":11262,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.802050+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:22.802050+0000","last_peered":"2024-09-18T11:59:22.802050+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:22.802050+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33838290,"num_objects":8263,"num_object_clones":0,"num_object_copies":16526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11183,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4731},{"shards":"0,2","objects":2074}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11249","reported_seq":11307,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:23.072997+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:23.072997+0000","last_peered":"2024-09-18T11:59:23.072997+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:23.072997+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33645778,"num_objects":8216,"num_object_clones":0,"num_object_copies":16432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11247,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2035},{"shards":"2","objects":4666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11220","reported_seq":11732,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:23.096973+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:23.096973+0000","last_peered":"2024-09-18T11:59:23.096973+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:23.096973+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33906688,"num_objects":8278,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1320,"num_objects_misplaced":2123,"num_objects_unfound":0,"num_objects_dirty":8278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7700,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2123},{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11251","reported_seq":11335,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.844217+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:22.844217+0000","last_peered":"2024-09-18T11:59:22.844217+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:22.844217+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33903209,"num_objects":8278,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11250,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4743},{"shards":"0,2","objects":2049}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11127","reported_seq":18275,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:18.249405+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:18.249405+0000","last_peered":"2024-09-18T11:59:18.249405+0000","last_clean":"2024-09-18T11:59:18.249405+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:18.249405+0000","last_undegraded":"2024-09-18T11:59:18.249405+0000","last_fullsized":"2024-09-18T11:59:18.249405+0000","mapping_epoch":40,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34387771,"num_objects":8398,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11124,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11076","reported_seq":11159,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.792958+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:22.792958+0000","last_peered":"2024-09-18T11:59:22.792958+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:22.792958+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33268946,"num_objects":8124,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11074,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4531},{"shards":"0,2","objects":2118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11218","reported_seq":11305,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.816942+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:22.816942+0000","last_peered":"2024-09-18T11:59:22.816942+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:22.816942+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33974692,"num_objects":8298,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11215,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4775},{"shards":"0,2","objects":2066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":541868108,"num_objects":132310,"num_object_clones":0,"num_object_copies":264620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33289,"num_objects_misplaced":2123,"num_objects_unfound":0,"num_objects_dirty":132310,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162905,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7178,"num_bytes_recovered":29299305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171562496,"data_stored":1171484772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160612,"ondisk_log_size":160612,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051650,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684304,"kb_used_data":378120,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93687536,"statfs":{"total":96636764160,"available":95936036864,"internally_reserved":0,"allocated":387194880,"data_stored":386956764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.189},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.28100000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.252},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.223}]}]},{"osd":1,"up_from":35,"seq":150323855412,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616240,"kb_used_data":314212,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93755600,"statfs":{"total":96636764160,"available":96005734400,"internally_reserved":0,"allocated":321753088,"data_stored":321452207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,1,1,2,2,3,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.255},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.28799999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":4.819},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":4.7560000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751184,"kb_used_data":453384,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93620656,"statfs":{"total":96636764160,"available":95867551744,"internally_reserved":0,"allocated":464265216,"data_stored":464027550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,0,2,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.30099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":463437824,"data_stored":463434345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":321368064,"data_stored":321301370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386756608,"data_stored":386749057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:24.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:59:24.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'11210", 'reported_seq': 11283, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:22.842872+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:59:22.842872+0000', 'last_peered': '2024-09-18T11:59:22.842872+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:59:22.842872+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'1200", 'ondisk_log_start': "30'1200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34103296, 'num_objects': 8326, 'num_object_clones': 0, 'num_object_copies': 16652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11210, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4727}, {'shards': '0,2', 'objects': 2157}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:24.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:59:24.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'11205", 'reported_seq': 11393, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:23.063356+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:59:23.063356+0000', 'last_peered': '2024-09-18T11:59:23.063356+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:59:23.063356+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'1200", 'ondisk_log_start': "30'1200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34209816, 'num_objects': 8353, 'num_object_clones': 0, 'num_object_copies': 16706, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8353, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8063, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2111}, {'shards': '2', 'objects': 4817}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:24.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:59:24.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'11213", 'reported_seq': 11295, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:22.810585+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:59:22.810585+0000', 'last_peered': '2024-09-18T11:59:22.810585+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:59:22.810585+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'1200", 'ondisk_log_start': "30'1200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33629394, 'num_objects': 8212, 'num_object_clones': 0, 'num_object_copies': 16424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8212, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11211, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4685}, {'shards': '0,2', 'objects': 2028}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:24.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:59:24.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'11117", 'reported_seq': 11206, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:22.652644+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:59:22.652644+0000', 'last_peered': '2024-09-18T11:59:22.652644+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:59:22.652644+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'1100", 'ondisk_log_start': "30'1100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33813714, 'num_objects': 8257, 'num_object_clones': 0, 'num_object_copies': 16514, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11115, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4698}, {'shards': '0,2', 'objects': 2131}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:24.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:59:24.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'11185", 'reported_seq': 11262, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:22.802050+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:59:22.802050+0000', 'last_peered': '2024-09-18T11:59:22.802050+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:59:22.802050+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'1100", 'ondisk_log_start': "30'1100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33838290, 'num_objects': 8263, 'num_object_clones': 0, 'num_object_copies': 16526, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11183, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4731}, {'shards': '0,2', 'objects': 2074}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:24.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:59:24.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'11249", 'reported_seq': 11307, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:23.072997+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:59:23.072997+0000', 'last_peered': '2024-09-18T11:59:23.072997+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:59:23.072997+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'1200", 'ondisk_log_start': "30'1200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33645778, 'num_objects': 8216, 'num_object_clones': 0, 'num_object_copies': 16432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11247, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2035}, {'shards': '2', 'objects': 4666}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:24.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:59:24.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'11220", 'reported_seq': 11732, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:23.096973+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T11:59:23.096973+0000', 'last_peered': '2024-09-18T11:59:23.096973+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:59:23.096973+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'1200", 'ondisk_log_start': "30'1200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33906688, 'num_objects': 8278, 'num_object_clones': 0, 'num_object_copies': 16556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1320, 'num_objects_misplaced': 2123, 'num_objects_unfound': 0, 'num_objects_dirty': 8278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7700, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 150, 'num_bytes_recovered': 614400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2123}, {'shards': '2', 'objects': 4835}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:24.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:59:24.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'11251", 'reported_seq': 11335, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:22.844217+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:59:22.844217+0000', 'last_peered': '2024-09-18T11:59:22.844217+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:59:22.844217+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'1200", 'ondisk_log_start': "30'1200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33903209, 'num_objects': 8278, 'num_object_clones': 0, 'num_object_copies': 16556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11250, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4743}, {'shards': '0,2', 'objects': 2049}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:24.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:59:24.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'11076", 'reported_seq': 11159, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:22.792958+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:59:22.792958+0000', 'last_peered': '2024-09-18T11:59:22.792958+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:59:22.792958+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'1000", 'ondisk_log_start': "30'1000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33268946, 'num_objects': 8124, 'num_object_clones': 0, 'num_object_copies': 16248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11074, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4531}, {'shards': '0,2', 'objects': 2118}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:24.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:59:24.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'11218", 'reported_seq': 11305, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:22.816942+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:59:22.816942+0000', 'last_peered': '2024-09-18T11:59:22.816942+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:59:22.816942+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'1200", 'ondisk_log_start': "30'1200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33974692, 'num_objects': 8298, 'num_object_clones': 0, 'num_object_copies': 16596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11215, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4775}, {'shards': '0,2', 'objects': 2066}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:24.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:59:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:25.414 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33289/264156 objects degraded (12.602%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":365,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":132078,"data_bytes":540909663,"bytes_used":2104270848,"bytes_avail":287806021632,"bytes_total":289910292480,"degraded_objects":33289,"degraded_total":264156,"degraded_ratio":0.12602023046987387,"misplaced_objects":2123,"misplaced_total":264156,"misplaced_ratio":0.0080369175790063439,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":118501,"recovering_keys_per_sec":0,"num_objects_recovered":273,"num_bytes_recovered":1118208,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":434},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:59:25.465 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:59:25.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:26.314 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:26.327 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-09-18T11:59:25.140210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":540909663,"num_objects":132078,"num_object_clones":0,"num_object_copies":264156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33289,"num_objects_misplaced":2123,"num_objects_unfound":0,"num_objects_dirty":132078,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163141,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7179,"num_bytes_recovered":29299324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160649,"ondisk_log_size":160649,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2054952,"kb_used_data":1144784,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281060568,"statfs":{"total":289910292480,"available":287806021632,"internally_reserved":0,"allocated":1172258816,"data_stored":1171482153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,3,6,5,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16801792,"num_objects":-4102,"num_object_clones":0,"num_object_copies":-8204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2275,"num_objects_misplaced":2000,"num_objects_unfound":0,"num_objects_dirty":-4102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4102,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":1118208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.436224"},"pg_stats":[{"pgid":"2.f","version":"41'11210","reported_seq":11283,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.842872+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:22.842872+0000","last_peered":"2024-09-18T11:59:22.842872+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:22.842872+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34103296,"num_objects":8326,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11210,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4727},{"shards":"0,2","objects":2157}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11192","reported_seq":11352,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:23.102077+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:23.102077+0000","last_peered":"2024-09-18T11:59:23.102077+0000","last_clean":"2024-09-18T11:59:23.102077+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:23.102077+0000","last_undegraded":"2024-09-18T11:59:23.102077+0000","last_fullsized":"2024-09-18T11:59:23.102077+0000","mapping_epoch":29,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":8240,"num_object_clones":0,"num_object_copies":16480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8384,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11205","reported_seq":11393,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:23.063356+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:23.063356+0000","last_peered":"2024-09-18T11:59:23.063356+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:23.063356+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209816,"num_objects":8353,"num_object_clones":0,"num_object_copies":16706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8353,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8063,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2111},{"shards":"2","objects":4817}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11112","reported_seq":11143,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:22.956624+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:22.956624+0000","last_peered":"2024-09-18T11:59:22.956624+0000","last_clean":"2024-09-18T11:59:22.956624+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:22.956624+0000","last_undegraded":"2024-09-18T11:59:22.956624+0000","last_fullsized":"2024-09-18T11:59:22.956624+0000","mapping_epoch":29,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33374208,"num_objects":8148,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11112,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11155","reported_seq":11348,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:22.780969+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:22.780969+0000","last_peered":"2024-09-18T11:59:22.780969+0000","last_clean":"2024-09-18T11:59:22.780969+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:22.780969+0000","last_undegraded":"2024-09-18T11:59:22.780969+0000","last_fullsized":"2024-09-18T11:59:22.780969+0000","mapping_epoch":29,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33964032,"num_objects":8292,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8320,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11213","reported_seq":11295,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.810585+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:22.810585+0000","last_peered":"2024-09-18T11:59:22.810585+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:22.810585+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33629394,"num_objects":8212,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11211,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4685},{"shards":"0,2","objects":2028}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11117","reported_seq":11206,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.652644+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:22.652644+0000","last_peered":"2024-09-18T11:59:22.652644+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:22.652644+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33813714,"num_objects":8257,"num_object_clones":0,"num_object_copies":16514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11115,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4698},{"shards":"0,2","objects":2131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11328","reported_seq":11555,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:22.775071+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:22.775071+0000","last_peered":"2024-09-18T11:59:22.775071+0000","last_clean":"2024-09-18T11:59:22.775071+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:22.775071+0000","last_undegraded":"2024-09-18T11:59:22.775071+0000","last_fullsized":"2024-09-18T11:59:22.775071+0000","mapping_epoch":29,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33890304,"num_objects":8274,"num_object_clones":0,"num_object_copies":16548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11164","reported_seq":18294,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:23.315771+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:23.315771+0000","last_peered":"2024-09-18T11:59:23.315771+0000","last_clean":"2024-09-18T11:59:23.315771+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:23.315771+0000","last_undegraded":"2024-09-18T11:59:23.315771+0000","last_fullsized":"2024-09-18T11:59:23.315771+0000","mapping_epoch":38,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33756370,"num_objects":8243,"num_object_clones":0,"num_object_copies":16486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11162,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11185","reported_seq":11262,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.802050+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:22.802050+0000","last_peered":"2024-09-18T11:59:22.802050+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:22.802050+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1100","ondisk_log_start":"30'1100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33838290,"num_objects":8263,"num_object_clones":0,"num_object_copies":16526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11183,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4731},{"shards":"0,2","objects":2074}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11249","reported_seq":11307,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:23.072997+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:23.072997+0000","last_peered":"2024-09-18T11:59:23.072997+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:23.072997+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33645778,"num_objects":8216,"num_object_clones":0,"num_object_copies":16432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11247,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2035},{"shards":"2","objects":4666}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11220","reported_seq":11732,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:23.096973+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:23.096973+0000","last_peered":"2024-09-18T11:59:23.096973+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:23.096973+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33906688,"num_objects":8278,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1320,"num_objects_misplaced":2123,"num_objects_unfound":0,"num_objects_dirty":8278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7700,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2123},{"shards":"2","objects":4835}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11251","reported_seq":11335,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.844217+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:22.844217+0000","last_peered":"2024-09-18T11:59:22.844217+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:22.844217+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33903209,"num_objects":8278,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11250,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4743},{"shards":"0,2","objects":2049}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11251","reported_seq":18399,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:23.273892+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:23.273892+0000","last_peered":"2024-09-18T11:59:23.273892+0000","last_clean":"2024-09-18T11:59:23.273892+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:23.273892+0000","last_undegraded":"2024-09-18T11:59:23.273892+0000","last_fullsized":"2024-09-18T11:59:23.273892+0000","mapping_epoch":40,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33879867,"num_objects":8274,"num_object_clones":0,"num_object_copies":16548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11248,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11076","reported_seq":11159,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.792958+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:22.792958+0000","last_peered":"2024-09-18T11:59:22.792958+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:22.792958+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1000","ondisk_log_start":"30'1000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33268946,"num_objects":8124,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11074,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4531},{"shards":"0,2","objects":2118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11218","reported_seq":11305,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:22.816942+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:22.816942+0000","last_peered":"2024-09-18T11:59:22.816942+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:22.816942+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33974692,"num_objects":8298,"num_object_clones":0,"num_object_copies":16596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11215,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4775},{"shards":"0,2","objects":2066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":540909644,"num_objects":132076,"num_object_clones":0,"num_object_copies":264152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33289,"num_objects_misplaced":2123,"num_objects_unfound":0,"num_objects_dirty":132076,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163139,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7178,"num_bytes_recovered":29299305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1170595840,"data_stored":1170518116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160646,"ondisk_log_size":160646,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051650,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684304,"kb_used_data":378120,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93687536,"statfs":{"total":96636764160,"available":95936036864,"internally_reserved":0,"allocated":387194880,"data_stored":386956764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.59399999999999997,"15min":0.45000000000000001},"min":{"1min":0.182,"5min":0.153,"15min":0.153},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.189},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0229999999999999,"15min":0.53500000000000003},"min":{"1min":0.187,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.795999999999999,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.28100000000000003}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.252},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.223}]}]},{"osd":1,"up_from":35,"seq":150323855413,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619464,"kb_used_data":313280,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93752376,"statfs":{"total":96636764160,"available":96002433024,"internally_reserved":0,"allocated":320798720,"data_stored":320497839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":6.1920000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.41899999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751184,"kb_used_data":453384,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93620656,"statfs":{"total":96636764160,"available":95867551744,"internally_reserved":0,"allocated":464265216,"data_stored":464027550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,0,2,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.30099999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":0.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":463437824,"data_stored":463434345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":320401408,"data_stored":320334714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386756608,"data_stored":386749057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:29.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:30.194 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:30.234 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:30.243 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-18T11:59:29.141074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":532312159,"num_objects":129979,"num_object_clones":0,"num_object_copies":259958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33381,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":129979,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165240,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7246,"num_bytes_recovered":29573756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2060216,"kb_used_data":1133408,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281055304,"statfs":{"total":289910292480,"available":287800631296,"internally_reserved":0,"allocated":1160609792,"data_stored":1159833129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,6,8,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25399296,"num_objects":-6201,"num_object_clones":0,"num_object_copies":-12402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2183,"num_objects_misplaced":1841,"num_objects_unfound":0,"num_objects_dirty":-6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6201,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":340,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002789"},"pg_stats":[{"pgid":"2.f","version":"41'11326","reported_seq":11399,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.826664+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:27.826664+0000","last_peered":"2024-09-18T11:59:27.826664+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:27.826664+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33628160,"num_objects":8210,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4611},{"shards":"0,2","objects":2041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11313","reported_seq":11473,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:28.054671+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:28.054671+0000","last_peered":"2024-09-18T11:59:28.054671+0000","last_clean":"2024-09-18T11:59:28.054671+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:28.054671+0000","last_undegraded":"2024-09-18T11:59:28.054671+0000","last_fullsized":"2024-09-18T11:59:28.054671+0000","mapping_epoch":29,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33255424,"num_objects":8119,"num_object_clones":0,"num_object_copies":16238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8505,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11319","reported_seq":11506,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:28.060795+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:28.060795+0000","last_peered":"2024-09-18T11:59:28.060795+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:28.060795+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33742872,"num_objects":8239,"num_object_clones":0,"num_object_copies":16478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8177,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1997},{"shards":"2","objects":4703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11244","reported_seq":11275,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:28.012309+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:28.012309+0000","last_peered":"2024-09-18T11:59:28.012309+0000","last_clean":"2024-09-18T11:59:28.012309+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:28.012309+0000","last_undegraded":"2024-09-18T11:59:28.012309+0000","last_fullsized":"2024-09-18T11:59:28.012309+0000","mapping_epoch":29,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32833536,"num_objects":8016,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11244,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11295","reported_seq":11488,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:27.837513+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:27.837513+0000","last_peered":"2024-09-18T11:59:27.837513+0000","last_clean":"2024-09-18T11:59:27.837513+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:27.837513+0000","last_undegraded":"2024-09-18T11:59:27.837513+0000","last_fullsized":"2024-09-18T11:59:27.837513+0000","mapping_epoch":29,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33390592,"num_objects":8152,"num_object_clones":0,"num_object_copies":16304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8460,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11332","reported_seq":11415,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.829014+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:27.829014+0000","last_peered":"2024-09-18T11:59:27.829014+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:27.829014+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33141970,"num_objects":8093,"num_object_clones":0,"num_object_copies":16186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11330,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4566},{"shards":"0,2","objects":1909}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11261","reported_seq":11349,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.870443+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:27.870443+0000","last_peered":"2024-09-18T11:59:27.870443+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:27.870443+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33223890,"num_objects":8113,"num_object_clones":0,"num_object_copies":16226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11259,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4554},{"shards":"0,2","objects":1987}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11466","reported_seq":11693,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:27.906041+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:27.906041+0000","last_peered":"2024-09-18T11:59:27.906041+0000","last_clean":"2024-09-18T11:59:27.906041+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:27.906041+0000","last_undegraded":"2024-09-18T11:59:27.906041+0000","last_fullsized":"2024-09-18T11:59:27.906041+0000","mapping_epoch":29,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33325056,"num_objects":8136,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7783,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11296","reported_seq":18426,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:28.355300+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:28.355300+0000","last_peered":"2024-09-18T11:59:28.355300+0000","last_clean":"2024-09-18T11:59:28.355300+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:28.355300+0000","last_undegraded":"2024-09-18T11:59:28.355300+0000","last_fullsized":"2024-09-18T11:59:28.355300+0000","mapping_epoch":38,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33215698,"num_objects":8111,"num_object_clones":0,"num_object_copies":16222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11314","reported_seq":11394,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.880470+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:27.880470+0000","last_peered":"2024-09-18T11:59:27.880470+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:27.880470+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33309906,"num_objects":8134,"num_object_clones":0,"num_object_copies":16268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11312,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4602},{"shards":"0,2","objects":1945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11397","reported_seq":11455,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:28.159799+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:28.159799+0000","last_peered":"2024-09-18T11:59:28.159799+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:28.159799+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33039570,"num_objects":8068,"num_object_clones":0,"num_object_copies":16136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1887},{"shards":"2","objects":4518}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11379","reported_seq":12025,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:28.160610+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:28.160610+0000","last_peered":"2024-09-18T11:59:28.160610+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:28.160610+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33255424,"num_objects":8119,"num_object_clones":0,"num_object_copies":16238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":8119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7859,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":888832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1964},{"shards":"2","objects":4743}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11372","reported_seq":11456,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.853727+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:27.853727+0000","last_peered":"2024-09-18T11:59:27.853727+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:27.853727+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33407593,"num_objects":8157,"num_object_clones":0,"num_object_copies":16314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11371,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4622},{"shards":"0,2","objects":1928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11383","reported_seq":18529,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:28.378726+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:28.378726+0000","last_peered":"2024-09-18T11:59:28.378726+0000","last_clean":"2024-09-18T11:59:28.378726+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:28.378726+0000","last_undegraded":"2024-09-18T11:59:28.378726+0000","last_fullsized":"2024-09-18T11:59:28.378726+0000","mapping_epoch":40,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33339195,"num_objects":8142,"num_object_clones":0,"num_object_copies":16284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11380,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11221","reported_seq":11304,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.857477+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:27.857477+0000","last_peered":"2024-09-18T11:59:27.857477+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:27.857477+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32675026,"num_objects":7979,"num_object_clones":0,"num_object_copies":15958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11219,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4386},{"shards":"0,2","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11327","reported_seq":11414,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.748507+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:27.748507+0000","last_peered":"2024-09-18T11:59:27.748507+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:27.748507+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528228,"num_objects":8189,"num_object_clones":0,"num_object_copies":16378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11324,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4666},{"shards":"0,2","objects":1957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":532312140,"num_objects":129977,"num_object_clones":0,"num_object_copies":259954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33381,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":129977,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165238,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7245,"num_bytes_recovered":29573737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1158729728,"data_stored":1158652004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051651,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684172,"kb_used_data":373828,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93687668,"statfs":{"total":96636764160,"available":95936172032,"internally_reserved":0,"allocated":382799872,"data_stored":382561756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":5.3559999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":5.4009999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.249}]}]},{"osd":1,"up_from":35,"seq":150323855414,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622608,"kb_used_data":312264,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93749232,"statfs":{"total":96636764160,"available":95999213568,"internally_reserved":0,"allocated":319758336,"data_stored":319457455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.371},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.39800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.34300000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753436,"kb_used_data":447316,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93618404,"statfs":{"total":96636764160,"available":95865245696,"internally_reserved":0,"allocated":458051584,"data_stored":457813918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.2910000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.23200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":8.2829999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":457097216,"data_stored":457093737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319324160,"data_stored":319257466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382308352,"data_stored":382300801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:30.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:31.105 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:31.119 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:31.155 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-18T11:59:29.141074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":532312159,"num_objects":129979,"num_object_clones":0,"num_object_copies":259958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33381,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":129979,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165240,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7246,"num_bytes_recovered":29573756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2060216,"kb_used_data":1133408,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281055304,"statfs":{"total":289910292480,"available":287800631296,"internally_reserved":0,"allocated":1160609792,"data_stored":1159833129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,6,8,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25399296,"num_objects":-6201,"num_object_clones":0,"num_object_copies":-12402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2183,"num_objects_misplaced":1841,"num_objects_unfound":0,"num_objects_dirty":-6201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6201,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":340,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002789"},"pg_stats":[{"pgid":"2.f","version":"41'11326","reported_seq":11399,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.826664+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:27.826664+0000","last_peered":"2024-09-18T11:59:27.826664+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:27.826664+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33628160,"num_objects":8210,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4611},{"shards":"0,2","objects":2041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11313","reported_seq":11473,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:28.054671+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:28.054671+0000","last_peered":"2024-09-18T11:59:28.054671+0000","last_clean":"2024-09-18T11:59:28.054671+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:28.054671+0000","last_undegraded":"2024-09-18T11:59:28.054671+0000","last_fullsized":"2024-09-18T11:59:28.054671+0000","mapping_epoch":29,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33255424,"num_objects":8119,"num_object_clones":0,"num_object_copies":16238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8505,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11319","reported_seq":11506,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:28.060795+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:28.060795+0000","last_peered":"2024-09-18T11:59:28.060795+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:28.060795+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33742872,"num_objects":8239,"num_object_clones":0,"num_object_copies":16478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8177,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1997},{"shards":"2","objects":4703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11244","reported_seq":11275,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:28.012309+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:28.012309+0000","last_peered":"2024-09-18T11:59:28.012309+0000","last_clean":"2024-09-18T11:59:28.012309+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:28.012309+0000","last_undegraded":"2024-09-18T11:59:28.012309+0000","last_fullsized":"2024-09-18T11:59:28.012309+0000","mapping_epoch":29,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32833536,"num_objects":8016,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11244,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11295","reported_seq":11488,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:27.837513+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:27.837513+0000","last_peered":"2024-09-18T11:59:27.837513+0000","last_clean":"2024-09-18T11:59:27.837513+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:27.837513+0000","last_undegraded":"2024-09-18T11:59:27.837513+0000","last_fullsized":"2024-09-18T11:59:27.837513+0000","mapping_epoch":29,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33390592,"num_objects":8152,"num_object_clones":0,"num_object_copies":16304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8460,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11332","reported_seq":11415,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.829014+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:27.829014+0000","last_peered":"2024-09-18T11:59:27.829014+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:27.829014+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33141970,"num_objects":8093,"num_object_clones":0,"num_object_copies":16186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11330,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4566},{"shards":"0,2","objects":1909}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11261","reported_seq":11349,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.870443+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:27.870443+0000","last_peered":"2024-09-18T11:59:27.870443+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:27.870443+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33223890,"num_objects":8113,"num_object_clones":0,"num_object_copies":16226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11259,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4554},{"shards":"0,2","objects":1987}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11466","reported_seq":11693,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:27.906041+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:27.906041+0000","last_peered":"2024-09-18T11:59:27.906041+0000","last_clean":"2024-09-18T11:59:27.906041+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:27.906041+0000","last_undegraded":"2024-09-18T11:59:27.906041+0000","last_fullsized":"2024-09-18T11:59:27.906041+0000","mapping_epoch":29,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33325056,"num_objects":8136,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7783,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11296","reported_seq":18426,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:28.355300+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:28.355300+0000","last_peered":"2024-09-18T11:59:28.355300+0000","last_clean":"2024-09-18T11:59:28.355300+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:28.355300+0000","last_undegraded":"2024-09-18T11:59:28.355300+0000","last_fullsized":"2024-09-18T11:59:28.355300+0000","mapping_epoch":38,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33215698,"num_objects":8111,"num_object_clones":0,"num_object_copies":16222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11314","reported_seq":11394,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.880470+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:27.880470+0000","last_peered":"2024-09-18T11:59:27.880470+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:27.880470+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33309906,"num_objects":8134,"num_object_clones":0,"num_object_copies":16268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11312,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4602},{"shards":"0,2","objects":1945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11397","reported_seq":11455,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:28.159799+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:28.159799+0000","last_peered":"2024-09-18T11:59:28.159799+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:28.159799+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33039570,"num_objects":8068,"num_object_clones":0,"num_object_copies":16136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1887},{"shards":"2","objects":4518}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11379","reported_seq":12025,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:28.160610+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:28.160610+0000","last_peered":"2024-09-18T11:59:28.160610+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:28.160610+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33255424,"num_objects":8119,"num_object_clones":0,"num_object_copies":16238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":8119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7859,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":888832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1964},{"shards":"2","objects":4743}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11372","reported_seq":11456,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.853727+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:27.853727+0000","last_peered":"2024-09-18T11:59:27.853727+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:27.853727+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33407593,"num_objects":8157,"num_object_clones":0,"num_object_copies":16314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11371,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4622},{"shards":"0,2","objects":1928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11383","reported_seq":18529,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:28.378726+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:28.378726+0000","last_peered":"2024-09-18T11:59:28.378726+0000","last_clean":"2024-09-18T11:59:28.378726+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:28.378726+0000","last_undegraded":"2024-09-18T11:59:28.378726+0000","last_fullsized":"2024-09-18T11:59:28.378726+0000","mapping_epoch":40,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33339195,"num_objects":8142,"num_object_clones":0,"num_object_copies":16284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11380,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11221","reported_seq":11304,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.857477+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:27.857477+0000","last_peered":"2024-09-18T11:59:27.857477+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:27.857477+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32675026,"num_objects":7979,"num_object_clones":0,"num_object_copies":15958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11219,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4386},{"shards":"0,2","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11327","reported_seq":11414,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.748507+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:27.748507+0000","last_peered":"2024-09-18T11:59:27.748507+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:27.748507+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528228,"num_objects":8189,"num_object_clones":0,"num_object_copies":16378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11324,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4666},{"shards":"0,2","objects":1957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":532312140,"num_objects":129977,"num_object_clones":0,"num_object_copies":259954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33381,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":129977,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165238,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7245,"num_bytes_recovered":29573737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1158729728,"data_stored":1158652004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051651,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684172,"kb_used_data":373828,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93687668,"statfs":{"total":96636764160,"available":95936172032,"internally_reserved":0,"allocated":382799872,"data_stored":382561756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":5.3559999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":5.4009999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.249}]}]},{"osd":1,"up_from":35,"seq":150323855414,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622608,"kb_used_data":312264,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93749232,"statfs":{"total":96636764160,"available":95999213568,"internally_reserved":0,"allocated":319758336,"data_stored":319457455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.371},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.39800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.34300000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753436,"kb_used_data":447316,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93618404,"statfs":{"total":96636764160,"available":95865245696,"internally_reserved":0,"allocated":458051584,"data_stored":457813918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.2910000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.23200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":8.2829999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":457097216,"data_stored":457093737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319324160,"data_stored":319257466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382308352,"data_stored":382300801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:31.156 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:59:31.156 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'11326", 'reported_seq': 11399, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:27.826664+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:59:27.826664+0000', 'last_peered': '2024-09-18T11:59:27.826664+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:59:27.826664+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'1300", 'ondisk_log_start': "30'1300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33628160, 'num_objects': 8210, 'num_object_clones': 0, 'num_object_copies': 16420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11326, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4611}, {'shards': '0,2', 'objects': 2041}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:31.156 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:59:31.157 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'11319", 'reported_seq': 11506, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:28.060795+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:59:28.060795+0000', 'last_peered': '2024-09-18T11:59:28.060795+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:59:28.060795+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'1300", 'ondisk_log_start': "30'1300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33742872, 'num_objects': 8239, 'num_object_clones': 0, 'num_object_copies': 16478, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8239, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8177, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1997}, {'shards': '2', 'objects': 4703}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:31.157 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:59:31.157 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'11332", 'reported_seq': 11415, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:27.829014+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:59:27.829014+0000', 'last_peered': '2024-09-18T11:59:27.829014+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:59:27.829014+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'1300", 'ondisk_log_start': "30'1300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33141970, 'num_objects': 8093, 'num_object_clones': 0, 'num_object_copies': 16186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11330, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4566}, {'shards': '0,2', 'objects': 1909}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:31.157 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:59:31.157 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'11261", 'reported_seq': 11349, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:27.870443+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:59:27.870443+0000', 'last_peered': '2024-09-18T11:59:27.870443+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:59:27.870443+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'1200", 'ondisk_log_start': "30'1200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33223890, 'num_objects': 8113, 'num_object_clones': 0, 'num_object_copies': 16226, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11259, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4554}, {'shards': '0,2', 'objects': 1987}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:31.157 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:59:31.157 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'11314", 'reported_seq': 11394, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:27.880470+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:59:27.880470+0000', 'last_peered': '2024-09-18T11:59:27.880470+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:59:27.880470+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'1300", 'ondisk_log_start': "30'1300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33309906, 'num_objects': 8134, 'num_object_clones': 0, 'num_object_copies': 16268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11312, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4602}, {'shards': '0,2', 'objects': 1945}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:31.157 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:59:31.157 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'11397", 'reported_seq': 11455, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:28.159799+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:59:28.159799+0000', 'last_peered': '2024-09-18T11:59:28.159799+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:59:28.159799+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'1300", 'ondisk_log_start': "30'1300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33039570, 'num_objects': 8068, 'num_object_clones': 0, 'num_object_copies': 16136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11395, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1887}, {'shards': '2', 'objects': 4518}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:31.158 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:59:31.158 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'11379", 'reported_seq': 12025, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:28.160610+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T11:59:28.160610+0000', 'last_peered': '2024-09-18T11:59:28.160610+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:59:28.160610+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'1300", 'ondisk_log_start': "30'1300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33255424, 'num_objects': 8119, 'num_object_clones': 0, 'num_object_copies': 16238, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1412, 'num_objects_misplaced': 1964, 'num_objects_unfound': 0, 'num_objects_dirty': 8119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7859, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 217, 'num_bytes_recovered': 888832, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1964}, {'shards': '2', 'objects': 4743}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:31.158 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:59:31.158 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'11372", 'reported_seq': 11456, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:27.853727+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:59:27.853727+0000', 'last_peered': '2024-09-18T11:59:27.853727+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:59:27.853727+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'1300", 'ondisk_log_start': "30'1300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33407593, 'num_objects': 8157, 'num_object_clones': 0, 'num_object_copies': 16314, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11371, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4622}, {'shards': '0,2', 'objects': 1928}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:31.158 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:59:31.158 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'11221", 'reported_seq': 11304, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:27.857477+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:59:27.857477+0000', 'last_peered': '2024-09-18T11:59:27.857477+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:59:27.857477+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'1200", 'ondisk_log_start': "30'1200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32675026, 'num_objects': 7979, 'num_object_clones': 0, 'num_object_copies': 15958, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11219, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4386}, {'shards': '0,2', 'objects': 1973}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:31.158 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:59:31.159 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'11327", 'reported_seq': 11414, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:27.748507+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:59:27.748507+0000', 'last_peered': '2024-09-18T11:59:27.748507+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:59:27.748507+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'1300", 'ondisk_log_start': "30'1300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33528228, 'num_objects': 8189, 'num_object_clones': 0, 'num_object_copies': 16378, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11324, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4666}, {'shards': '0,2', 'objects': 1957}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:31.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:59:32.107 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:32.107 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33381/259958 objects degraded (12.841%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":372,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":129979,"data_bytes":532312159,"bytes_used":2109661184,"bytes_avail":287800631296,"bytes_total":289910292480,"degraded_objects":33381,"degraded_total":259958,"degraded_ratio":0.12840920456381416,"misplaced_objects":1964,"misplaced_total":259958,"misplaced_ratio":0.0075550665876795483,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":53576,"recovering_keys_per_sec":0,"num_objects_recovered":157,"num_bytes_recovered":643072,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":348},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:59:32.158 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:59:32.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:33.009 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:33.024 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:33.053 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-18T11:59:31.141590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":532312159,"num_objects":129979,"num_object_clones":0,"num_object_copies":259958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33381,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":129979,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165240,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7246,"num_bytes_recovered":29573756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2060216,"kb_used_data":1133408,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281055304,"statfs":{"total":289910292480,"available":287800631296,"internally_reserved":0,"allocated":1160609792,"data_stored":1159833129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,6,8,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17113088,"num_objects":-4178,"num_object_clones":0,"num_object_copies":-8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":135,"num_objects_misplaced":-292,"num_objects_unfound":0,"num_objects_dirty":-4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4178,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":643072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002841"},"pg_stats":[{"pgid":"2.f","version":"41'11326","reported_seq":11399,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.826664+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:27.826664+0000","last_peered":"2024-09-18T11:59:27.826664+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:27.826664+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33628160,"num_objects":8210,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4611},{"shards":"0,2","objects":2041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11313","reported_seq":11473,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:28.054671+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:28.054671+0000","last_peered":"2024-09-18T11:59:28.054671+0000","last_clean":"2024-09-18T11:59:28.054671+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:28.054671+0000","last_undegraded":"2024-09-18T11:59:28.054671+0000","last_fullsized":"2024-09-18T11:59:28.054671+0000","mapping_epoch":29,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33255424,"num_objects":8119,"num_object_clones":0,"num_object_copies":16238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8505,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11319","reported_seq":11506,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:28.060795+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:28.060795+0000","last_peered":"2024-09-18T11:59:28.060795+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:28.060795+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33742872,"num_objects":8239,"num_object_clones":0,"num_object_copies":16478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8177,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1997},{"shards":"2","objects":4703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11244","reported_seq":11275,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:28.012309+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:28.012309+0000","last_peered":"2024-09-18T11:59:28.012309+0000","last_clean":"2024-09-18T11:59:28.012309+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:28.012309+0000","last_undegraded":"2024-09-18T11:59:28.012309+0000","last_fullsized":"2024-09-18T11:59:28.012309+0000","mapping_epoch":29,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32833536,"num_objects":8016,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11244,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11295","reported_seq":11488,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:27.837513+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:27.837513+0000","last_peered":"2024-09-18T11:59:27.837513+0000","last_clean":"2024-09-18T11:59:27.837513+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:27.837513+0000","last_undegraded":"2024-09-18T11:59:27.837513+0000","last_fullsized":"2024-09-18T11:59:27.837513+0000","mapping_epoch":29,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33390592,"num_objects":8152,"num_object_clones":0,"num_object_copies":16304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8460,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11332","reported_seq":11415,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.829014+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:27.829014+0000","last_peered":"2024-09-18T11:59:27.829014+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:27.829014+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33141970,"num_objects":8093,"num_object_clones":0,"num_object_copies":16186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11330,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4566},{"shards":"0,2","objects":1909}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11261","reported_seq":11349,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.870443+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:27.870443+0000","last_peered":"2024-09-18T11:59:27.870443+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:27.870443+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33223890,"num_objects":8113,"num_object_clones":0,"num_object_copies":16226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11259,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4554},{"shards":"0,2","objects":1987}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11466","reported_seq":11693,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:27.906041+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:27.906041+0000","last_peered":"2024-09-18T11:59:27.906041+0000","last_clean":"2024-09-18T11:59:27.906041+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:27.906041+0000","last_undegraded":"2024-09-18T11:59:27.906041+0000","last_fullsized":"2024-09-18T11:59:27.906041+0000","mapping_epoch":29,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33325056,"num_objects":8136,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7783,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11296","reported_seq":18426,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:28.355300+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:28.355300+0000","last_peered":"2024-09-18T11:59:28.355300+0000","last_clean":"2024-09-18T11:59:28.355300+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:28.355300+0000","last_undegraded":"2024-09-18T11:59:28.355300+0000","last_fullsized":"2024-09-18T11:59:28.355300+0000","mapping_epoch":38,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33215698,"num_objects":8111,"num_object_clones":0,"num_object_copies":16222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11314","reported_seq":11394,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.880470+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:27.880470+0000","last_peered":"2024-09-18T11:59:27.880470+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:27.880470+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33309906,"num_objects":8134,"num_object_clones":0,"num_object_copies":16268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11312,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4602},{"shards":"0,2","objects":1945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11397","reported_seq":11455,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:28.159799+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:28.159799+0000","last_peered":"2024-09-18T11:59:28.159799+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:28.159799+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33039570,"num_objects":8068,"num_object_clones":0,"num_object_copies":16136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1887},{"shards":"2","objects":4518}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11379","reported_seq":12025,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:28.160610+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:28.160610+0000","last_peered":"2024-09-18T11:59:28.160610+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:28.160610+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33255424,"num_objects":8119,"num_object_clones":0,"num_object_copies":16238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":8119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7859,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":888832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1964},{"shards":"2","objects":4743}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11372","reported_seq":11456,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.853727+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:27.853727+0000","last_peered":"2024-09-18T11:59:27.853727+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:27.853727+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33407593,"num_objects":8157,"num_object_clones":0,"num_object_copies":16314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11371,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4622},{"shards":"0,2","objects":1928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11383","reported_seq":18529,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:28.378726+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:28.378726+0000","last_peered":"2024-09-18T11:59:28.378726+0000","last_clean":"2024-09-18T11:59:28.378726+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:28.378726+0000","last_undegraded":"2024-09-18T11:59:28.378726+0000","last_fullsized":"2024-09-18T11:59:28.378726+0000","mapping_epoch":40,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33339195,"num_objects":8142,"num_object_clones":0,"num_object_copies":16284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11380,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11221","reported_seq":11304,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.857477+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:27.857477+0000","last_peered":"2024-09-18T11:59:27.857477+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:27.857477+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1200","ondisk_log_start":"30'1200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32675026,"num_objects":7979,"num_object_clones":0,"num_object_copies":15958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11219,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4386},{"shards":"0,2","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11327","reported_seq":11414,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:27.748507+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:27.748507+0000","last_peered":"2024-09-18T11:59:27.748507+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:27.748507+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33528228,"num_objects":8189,"num_object_clones":0,"num_object_copies":16378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11324,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4666},{"shards":"0,2","objects":1957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":532312140,"num_objects":129977,"num_object_clones":0,"num_object_copies":259954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33381,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":129977,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165238,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7245,"num_bytes_recovered":29573737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1158729728,"data_stored":1158652004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051651,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684172,"kb_used_data":373828,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93687668,"statfs":{"total":96636764160,"available":95936172032,"internally_reserved":0,"allocated":382799872,"data_stored":382561756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":5.3559999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":5.4009999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.249}]}]},{"osd":1,"up_from":35,"seq":150323855414,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622608,"kb_used_data":312264,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93749232,"statfs":{"total":96636764160,"available":95999213568,"internally_reserved":0,"allocated":319758336,"data_stored":319457455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.371},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.39800000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.34300000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753436,"kb_used_data":447316,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93618404,"statfs":{"total":96636764160,"available":95865245696,"internally_reserved":0,"allocated":458051584,"data_stored":457813918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.2910000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.23200000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.97099999999999997,"15min":0.63600000000000001},"min":{"1min":0.20100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":5.4829999999999997,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":0.82099999999999995,"15min":0.48799999999999999},"min":{"1min":0.22800000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.3680000000000003,"5min":9.4309999999999992,"15min":9.4309999999999992},"last":8.2829999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":457097216,"data_stored":457093737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319324160,"data_stored":319257466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382308352,"data_stored":382300801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:36.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:36.818 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:36.846 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:36.868 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-18T11:59:35.144255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":523788383,"num_objects":127898,"num_object_clones":0,"num_object_copies":255796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33435,"num_objects_misplaced":1825,"num_objects_unfound":0,"num_objects_dirty":127898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167321,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7331,"num_bytes_recovered":29921916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160629,"ondisk_log_size":160629,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2032004,"kb_used_data":1121828,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281083516,"statfs":{"total":289910292480,"available":287829520384,"internally_reserved":0,"allocated":1148751872,"data_stored":1147975209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,7,7,10,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18079744,"num_objects":-4414,"num_object_clones":0,"num_object_copies":-8828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":146,"num_objects_misplaced":-298,"num_objects_unfound":0,"num_objects_dirty":-4414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4414,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":622592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004515"},"pg_stats":[{"pgid":"2.f","version":"41'11449","reported_seq":11522,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.888286+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:32.888286+0000","last_peered":"2024-09-18T11:59:32.888286+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:32.888286+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33124352,"num_objects":8087,"num_object_clones":0,"num_object_copies":16174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11449,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4488},{"shards":"0,2","objects":1918}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11443","reported_seq":11603,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:33.068142+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:33.068142+0000","last_peered":"2024-09-18T11:59:33.068142+0000","last_clean":"2024-09-18T11:59:33.068142+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:33.068142+0000","last_undegraded":"2024-09-18T11:59:33.068142+0000","last_fullsized":"2024-09-18T11:59:33.068142+0000","mapping_epoch":29,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32722944,"num_objects":7989,"num_object_clones":0,"num_object_copies":15978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8635,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11466","reported_seq":11649,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:33.137424+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:33.137424+0000","last_peered":"2024-09-18T11:59:33.137424+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:33.137424+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33140760,"num_objects":8092,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8092,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8324,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1850},{"shards":"2","objects":4556}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11378","reported_seq":11409,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:33.015770+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:33.015770+0000","last_peered":"2024-09-18T11:59:33.015770+0000","last_clean":"2024-09-18T11:59:33.015770+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:33.015770+0000","last_undegraded":"2024-09-18T11:59:33.015770+0000","last_fullsized":"2024-09-18T11:59:33.015770+0000","mapping_epoch":29,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32284672,"num_objects":7882,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11378,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11433","reported_seq":11625,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:32.934273+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:32.934273+0000","last_peered":"2024-09-18T11:59:32.934273+0000","last_clean":"2024-09-18T11:59:32.934273+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:32.934273+0000","last_undegraded":"2024-09-18T11:59:32.934273+0000","last_fullsized":"2024-09-18T11:59:32.934273+0000","mapping_epoch":29,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32825344,"num_objects":8014,"num_object_clones":0,"num_object_copies":16028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8598,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11462","reported_seq":11545,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.863154+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:32.863154+0000","last_peered":"2024-09-18T11:59:32.863154+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:32.863154+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32609490,"num_objects":7963,"num_object_clones":0,"num_object_copies":15926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4436},{"shards":"0,2","objects":1779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11393","reported_seq":11482,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.809321+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:32.809321+0000","last_peered":"2024-09-18T11:59:32.809321+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:32.809321+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32683218,"num_objects":7981,"num_object_clones":0,"num_object_copies":15962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11391,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4422},{"shards":"0,2","objects":1855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11616","reported_seq":11843,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:32.927470+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:32.927470+0000","last_peered":"2024-09-18T11:59:32.927470+0000","last_clean":"2024-09-18T11:59:32.927470+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:32.927470+0000","last_undegraded":"2024-09-18T11:59:32.927470+0000","last_fullsized":"2024-09-18T11:59:32.927470+0000","mapping_epoch":29,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32710656,"num_objects":7986,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11404","reported_seq":18534,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:33.359646+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:33.359646+0000","last_peered":"2024-09-18T11:59:33.359646+0000","last_clean":"2024-09-18T11:59:33.359646+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:33.359646+0000","last_undegraded":"2024-09-18T11:59:33.359646+0000","last_fullsized":"2024-09-18T11:59:33.359646+0000","mapping_epoch":38,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32773330,"num_objects":8003,"num_object_clones":0,"num_object_copies":16006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11402,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11448","reported_seq":11528,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.904136+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:32.904136+0000","last_peered":"2024-09-18T11:59:32.904136+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:32.904136+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32761042,"num_objects":8000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11446,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4468},{"shards":"0,2","objects":1811}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11522","reported_seq":11580,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:33.117216+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:33.117216+0000","last_peered":"2024-09-18T11:59:33.117216+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:33.117216+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32527570,"num_objects":7943,"num_object_clones":0,"num_object_copies":15886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1762},{"shards":"2","objects":4393}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11511","reported_seq":12327,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:33.159701+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:33.159701+0000","last_peered":"2024-09-18T11:59:33.159701+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:33.159701+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32714752,"num_objects":7987,"num_object_clones":0,"num_object_copies":15974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1466,"num_objects_misplaced":1825,"num_objects_unfound":0,"num_objects_dirty":7987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7991,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1825},{"shards":"2","objects":4696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11512","reported_seq":11596,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.891956+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:32.891956+0000","last_peered":"2024-09-18T11:59:32.891956+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:32.891956+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32834153,"num_objects":8017,"num_object_clones":0,"num_object_copies":16034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11511,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4482},{"shards":"0,2","objects":1788}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11501","reported_seq":18649,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:33.377591+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:33.377591+0000","last_peered":"2024-09-18T11:59:33.377591+0000","last_clean":"2024-09-18T11:59:33.377591+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:33.377591+0000","last_undegraded":"2024-09-18T11:59:33.377591+0000","last_fullsized":"2024-09-18T11:59:33.377591+0000","mapping_epoch":40,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32855867,"num_objects":8024,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11498,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11335","reported_seq":11418,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.786095+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:32.786095+0000","last_peered":"2024-09-18T11:59:32.786095+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:32.786095+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32208082,"num_objects":7865,"num_object_clones":0,"num_object_copies":15730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11333,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4272},{"shards":"0,2","objects":1859}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11453","reported_seq":11540,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.929389+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:32.929389+0000","last_peered":"2024-09-18T11:59:32.929389+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:32.929389+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33012132,"num_objects":8063,"num_object_clones":0,"num_object_copies":16126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11450,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4540},{"shards":"0,2","objects":1831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":523788364,"num_objects":127896,"num_object_clones":0,"num_object_copies":255792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33435,"num_objects_misplaced":1825,"num_objects_unfound":0,"num_objects_dirty":127896,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167319,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7330,"num_bytes_recovered":29921897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1147015168,"data_stored":1146937444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160626,"ondisk_log_size":160626,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051652,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667452,"kb_used_data":369580,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93704388,"statfs":{"total":96636764160,"available":95953293312,"internally_reserved":0,"allocated":378449920,"data_stored":378211804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":5.6029999999999998},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.41199999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":5.7750000000000004},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.34100000000000003}]}]},{"osd":1,"up_from":35,"seq":150323855415,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613372,"kb_used_data":311348,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93758468,"statfs":{"total":96636764160,"available":96008671232,"internally_reserved":0,"allocated":318820352,"data_stored":318519471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.20000000000000001},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.433}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.23200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751180,"kb_used_data":440900,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93620660,"statfs":{"total":96636764160,"available":95867555840,"internally_reserved":0,"allocated":451481600,"data_stored":451243934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.308}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.35499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":450662400,"data_stored":450658921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":318418944,"data_stored":318352250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":377933824,"data_stored":377926273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:36.869 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:37.493 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:37.508 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-18T11:59:37.144724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":523788383,"num_objects":127898,"num_object_clones":0,"num_object_copies":255796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33435,"num_objects_misplaced":1825,"num_objects_unfound":0,"num_objects_dirty":127898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167321,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7331,"num_bytes_recovered":29921916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160629,"ondisk_log_size":160629,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2032004,"kb_used_data":1121828,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281083516,"statfs":{"total":289910292480,"available":287829520384,"internally_reserved":0,"allocated":1148751872,"data_stored":1147975209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,7,7,10,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17121280,"num_objects":-4180,"num_object_clones":0,"num_object_copies":-8360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":146,"num_objects_misplaced":-298,"num_objects_unfound":0,"num_objects_dirty":-4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":622592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004514"},"pg_stats":[{"pgid":"2.f","version":"41'11449","reported_seq":11522,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.888286+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:32.888286+0000","last_peered":"2024-09-18T11:59:32.888286+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:32.888286+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33124352,"num_objects":8087,"num_object_clones":0,"num_object_copies":16174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11449,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4488},{"shards":"0,2","objects":1918}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11443","reported_seq":11603,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:33.068142+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:33.068142+0000","last_peered":"2024-09-18T11:59:33.068142+0000","last_clean":"2024-09-18T11:59:33.068142+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:33.068142+0000","last_undegraded":"2024-09-18T11:59:33.068142+0000","last_fullsized":"2024-09-18T11:59:33.068142+0000","mapping_epoch":29,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32722944,"num_objects":7989,"num_object_clones":0,"num_object_copies":15978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8635,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11466","reported_seq":11649,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:33.137424+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:33.137424+0000","last_peered":"2024-09-18T11:59:33.137424+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:33.137424+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33140760,"num_objects":8092,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8092,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8324,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1850},{"shards":"2","objects":4556}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11378","reported_seq":11409,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:33.015770+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:33.015770+0000","last_peered":"2024-09-18T11:59:33.015770+0000","last_clean":"2024-09-18T11:59:33.015770+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:33.015770+0000","last_undegraded":"2024-09-18T11:59:33.015770+0000","last_fullsized":"2024-09-18T11:59:33.015770+0000","mapping_epoch":29,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32284672,"num_objects":7882,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11378,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11433","reported_seq":11625,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:32.934273+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:32.934273+0000","last_peered":"2024-09-18T11:59:32.934273+0000","last_clean":"2024-09-18T11:59:32.934273+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:32.934273+0000","last_undegraded":"2024-09-18T11:59:32.934273+0000","last_fullsized":"2024-09-18T11:59:32.934273+0000","mapping_epoch":29,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32825344,"num_objects":8014,"num_object_clones":0,"num_object_copies":16028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8598,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11462","reported_seq":11545,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.863154+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:32.863154+0000","last_peered":"2024-09-18T11:59:32.863154+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:32.863154+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32609490,"num_objects":7963,"num_object_clones":0,"num_object_copies":15926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4436},{"shards":"0,2","objects":1779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11393","reported_seq":11482,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.809321+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:32.809321+0000","last_peered":"2024-09-18T11:59:32.809321+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:32.809321+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32683218,"num_objects":7981,"num_object_clones":0,"num_object_copies":15962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11391,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4422},{"shards":"0,2","objects":1855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11616","reported_seq":11843,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:32.927470+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:32.927470+0000","last_peered":"2024-09-18T11:59:32.927470+0000","last_clean":"2024-09-18T11:59:32.927470+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:32.927470+0000","last_undegraded":"2024-09-18T11:59:32.927470+0000","last_fullsized":"2024-09-18T11:59:32.927470+0000","mapping_epoch":29,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32710656,"num_objects":7986,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11404","reported_seq":18534,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:33.359646+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:33.359646+0000","last_peered":"2024-09-18T11:59:33.359646+0000","last_clean":"2024-09-18T11:59:33.359646+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:33.359646+0000","last_undegraded":"2024-09-18T11:59:33.359646+0000","last_fullsized":"2024-09-18T11:59:33.359646+0000","mapping_epoch":38,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32773330,"num_objects":8003,"num_object_clones":0,"num_object_copies":16006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11402,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11448","reported_seq":11528,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.904136+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:32.904136+0000","last_peered":"2024-09-18T11:59:32.904136+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:32.904136+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32761042,"num_objects":8000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11446,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4468},{"shards":"0,2","objects":1811}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11522","reported_seq":11580,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:33.117216+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:33.117216+0000","last_peered":"2024-09-18T11:59:33.117216+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:33.117216+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32527570,"num_objects":7943,"num_object_clones":0,"num_object_copies":15886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1762},{"shards":"2","objects":4393}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11511","reported_seq":12327,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:33.159701+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:33.159701+0000","last_peered":"2024-09-18T11:59:33.159701+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:33.159701+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32714752,"num_objects":7987,"num_object_clones":0,"num_object_copies":15974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1466,"num_objects_misplaced":1825,"num_objects_unfound":0,"num_objects_dirty":7987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7991,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1825},{"shards":"2","objects":4696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11512","reported_seq":11596,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.891956+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:32.891956+0000","last_peered":"2024-09-18T11:59:32.891956+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:32.891956+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32834153,"num_objects":8017,"num_object_clones":0,"num_object_copies":16034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11511,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4482},{"shards":"0,2","objects":1788}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11501","reported_seq":18649,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:33.377591+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:33.377591+0000","last_peered":"2024-09-18T11:59:33.377591+0000","last_clean":"2024-09-18T11:59:33.377591+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:33.377591+0000","last_undegraded":"2024-09-18T11:59:33.377591+0000","last_fullsized":"2024-09-18T11:59:33.377591+0000","mapping_epoch":40,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32855867,"num_objects":8024,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11498,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11335","reported_seq":11418,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.786095+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:32.786095+0000","last_peered":"2024-09-18T11:59:32.786095+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:32.786095+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32208082,"num_objects":7865,"num_object_clones":0,"num_object_copies":15730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11333,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4272},{"shards":"0,2","objects":1859}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11453","reported_seq":11540,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.929389+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:32.929389+0000","last_peered":"2024-09-18T11:59:32.929389+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:32.929389+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33012132,"num_objects":8063,"num_object_clones":0,"num_object_copies":16126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11450,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4540},{"shards":"0,2","objects":1831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":523788364,"num_objects":127896,"num_object_clones":0,"num_object_copies":255792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33435,"num_objects_misplaced":1825,"num_objects_unfound":0,"num_objects_dirty":127896,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167319,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7330,"num_bytes_recovered":29921897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1147015168,"data_stored":1146937444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160626,"ondisk_log_size":160626,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051652,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667452,"kb_used_data":369580,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93704388,"statfs":{"total":96636764160,"available":95953293312,"internally_reserved":0,"allocated":378449920,"data_stored":378211804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":5.6029999999999998},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.41199999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":5.7750000000000004},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.34100000000000003}]}]},{"osd":1,"up_from":35,"seq":150323855415,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613372,"kb_used_data":311348,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93758468,"statfs":{"total":96636764160,"available":96008671232,"internally_reserved":0,"allocated":318820352,"data_stored":318519471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.20000000000000001},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.433}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.23200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751180,"kb_used_data":440900,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93620660,"statfs":{"total":96636764160,"available":95867555840,"internally_reserved":0,"allocated":451481600,"data_stored":451243934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.308}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.35499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":450662400,"data_stored":450658921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":318418944,"data_stored":318352250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":377933824,"data_stored":377926273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:37.510 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:59:37.510 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'11449", 'reported_seq': 11522, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:32.888286+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:59:32.888286+0000', 'last_peered': '2024-09-18T11:59:32.888286+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:59:32.888286+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'1400", 'ondisk_log_start': "30'1400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33124352, 'num_objects': 8087, 'num_object_clones': 0, 'num_object_copies': 16174, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11449, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4488}, {'shards': '0,2', 'objects': 1918}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:37.510 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:59:37.510 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'11466", 'reported_seq': 11649, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:33.137424+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:59:33.137424+0000', 'last_peered': '2024-09-18T11:59:33.137424+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:59:33.137424+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'1400", 'ondisk_log_start': "30'1400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33140760, 'num_objects': 8092, 'num_object_clones': 0, 'num_object_copies': 16184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8092, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8324, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1850}, {'shards': '2', 'objects': 4556}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:37.510 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:59:37.510 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'11462", 'reported_seq': 11545, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:32.863154+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:59:32.863154+0000', 'last_peered': '2024-09-18T11:59:32.863154+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:59:32.863154+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'1400", 'ondisk_log_start': "30'1400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32609490, 'num_objects': 7963, 'num_object_clones': 0, 'num_object_copies': 15926, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7963, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11460, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4436}, {'shards': '0,2', 'objects': 1779}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:37.510 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:59:37.510 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'11393", 'reported_seq': 11482, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:32.809321+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:59:32.809321+0000', 'last_peered': '2024-09-18T11:59:32.809321+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:59:32.809321+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'1300", 'ondisk_log_start': "30'1300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32683218, 'num_objects': 7981, 'num_object_clones': 0, 'num_object_copies': 15962, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11391, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4422}, {'shards': '0,2', 'objects': 1855}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:37.510 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:59:37.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'11448", 'reported_seq': 11528, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:32.904136+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:59:32.904136+0000', 'last_peered': '2024-09-18T11:59:32.904136+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:59:32.904136+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'1400", 'ondisk_log_start': "30'1400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32761042, 'num_objects': 8000, 'num_object_clones': 0, 'num_object_copies': 16000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8000, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11446, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4468}, {'shards': '0,2', 'objects': 1811}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:37.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:59:37.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'11522", 'reported_seq': 11580, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:33.117216+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:59:33.117216+0000', 'last_peered': '2024-09-18T11:59:33.117216+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:59:33.117216+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'1500", 'ondisk_log_start': "30'1500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32527570, 'num_objects': 7943, 'num_object_clones': 0, 'num_object_copies': 15886, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11520, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1762}, {'shards': '2', 'objects': 4393}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:37.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:59:37.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'11511", 'reported_seq': 12327, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:33.159701+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T11:59:33.159701+0000', 'last_peered': '2024-09-18T11:59:33.159701+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:59:33.159701+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'1500", 'ondisk_log_start': "30'1500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32714752, 'num_objects': 7987, 'num_object_clones': 0, 'num_object_copies': 15974, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1466, 'num_objects_misplaced': 1825, 'num_objects_unfound': 0, 'num_objects_dirty': 7987, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7991, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 302, 'num_bytes_recovered': 1236992, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1825}, {'shards': '2', 'objects': 4696}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:37.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:59:37.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'11512", 'reported_seq': 11596, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:32.891956+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:59:32.891956+0000', 'last_peered': '2024-09-18T11:59:32.891956+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:59:32.891956+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'1500", 'ondisk_log_start': "30'1500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32834153, 'num_objects': 8017, 'num_object_clones': 0, 'num_object_copies': 16034, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8017, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11511, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4482}, {'shards': '0,2', 'objects': 1788}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:37.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:59:37.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'11335", 'reported_seq': 11418, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:32.786095+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:59:32.786095+0000', 'last_peered': '2024-09-18T11:59:32.786095+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:59:32.786095+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'1300", 'ondisk_log_start': "30'1300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32208082, 'num_objects': 7865, 'num_object_clones': 0, 'num_object_copies': 15730, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11333, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4272}, {'shards': '0,2', 'objects': 1859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:37.511 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:59:37.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'11453", 'reported_seq': 11540, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:32.929389+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:59:32.929389+0000', 'last_peered': '2024-09-18T11:59:32.929389+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:59:32.929389+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'1400", 'ondisk_log_start': "30'1400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33012132, 'num_objects': 8063, 'num_object_clones': 0, 'num_object_copies': 16126, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8063, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11450, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4540}, {'shards': '0,2', 'objects': 1831}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:37.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:59:38.259 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:38.259 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33435/255796 objects degraded (13.071%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":378,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":127898,"data_bytes":523788383,"bytes_used":2080772096,"bytes_avail":287829520384,"bytes_total":289910292480,"degraded_objects":33435,"degraded_total":255796,"degraded_ratio":0.13070962798480038,"misplaced_objects":1825,"misplaced_total":255796,"misplaced_ratio":0.007134591627703326,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":51863,"recovering_keys_per_sec":0,"num_objects_recovered":152,"num_bytes_recovered":622592,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":348},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:59:38.307 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:59:38.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:38.975 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:38.998 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:39.020 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-18T11:59:37.144724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":523788383,"num_objects":127898,"num_object_clones":0,"num_object_copies":255796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33435,"num_objects_misplaced":1825,"num_objects_unfound":0,"num_objects_dirty":127898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167321,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7331,"num_bytes_recovered":29921916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160629,"ondisk_log_size":160629,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2032004,"kb_used_data":1121828,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281083516,"statfs":{"total":289910292480,"available":287829520384,"internally_reserved":0,"allocated":1148751872,"data_stored":1147975209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,7,7,10,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17121280,"num_objects":-4180,"num_object_clones":0,"num_object_copies":-8360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":146,"num_objects_misplaced":-298,"num_objects_unfound":0,"num_objects_dirty":-4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":622592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004514"},"pg_stats":[{"pgid":"2.f","version":"41'11449","reported_seq":11522,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.888286+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:32.888286+0000","last_peered":"2024-09-18T11:59:32.888286+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:32.888286+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33124352,"num_objects":8087,"num_object_clones":0,"num_object_copies":16174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11449,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4488},{"shards":"0,2","objects":1918}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11443","reported_seq":11603,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:33.068142+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:33.068142+0000","last_peered":"2024-09-18T11:59:33.068142+0000","last_clean":"2024-09-18T11:59:33.068142+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:33.068142+0000","last_undegraded":"2024-09-18T11:59:33.068142+0000","last_fullsized":"2024-09-18T11:59:33.068142+0000","mapping_epoch":29,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32722944,"num_objects":7989,"num_object_clones":0,"num_object_copies":15978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8635,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11466","reported_seq":11649,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:33.137424+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:33.137424+0000","last_peered":"2024-09-18T11:59:33.137424+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:33.137424+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33140760,"num_objects":8092,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8092,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8324,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1850},{"shards":"2","objects":4556}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11378","reported_seq":11409,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:33.015770+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:33.015770+0000","last_peered":"2024-09-18T11:59:33.015770+0000","last_clean":"2024-09-18T11:59:33.015770+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:33.015770+0000","last_undegraded":"2024-09-18T11:59:33.015770+0000","last_fullsized":"2024-09-18T11:59:33.015770+0000","mapping_epoch":29,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32284672,"num_objects":7882,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11378,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11433","reported_seq":11625,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:32.934273+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:32.934273+0000","last_peered":"2024-09-18T11:59:32.934273+0000","last_clean":"2024-09-18T11:59:32.934273+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:32.934273+0000","last_undegraded":"2024-09-18T11:59:32.934273+0000","last_fullsized":"2024-09-18T11:59:32.934273+0000","mapping_epoch":29,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32825344,"num_objects":8014,"num_object_clones":0,"num_object_copies":16028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8598,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11462","reported_seq":11545,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.863154+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:32.863154+0000","last_peered":"2024-09-18T11:59:32.863154+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:32.863154+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32609490,"num_objects":7963,"num_object_clones":0,"num_object_copies":15926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4436},{"shards":"0,2","objects":1779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11393","reported_seq":11482,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.809321+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:32.809321+0000","last_peered":"2024-09-18T11:59:32.809321+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:32.809321+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32683218,"num_objects":7981,"num_object_clones":0,"num_object_copies":15962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11391,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4422},{"shards":"0,2","objects":1855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11616","reported_seq":11843,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:32.927470+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:32.927470+0000","last_peered":"2024-09-18T11:59:32.927470+0000","last_clean":"2024-09-18T11:59:32.927470+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:32.927470+0000","last_undegraded":"2024-09-18T11:59:32.927470+0000","last_fullsized":"2024-09-18T11:59:32.927470+0000","mapping_epoch":29,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32710656,"num_objects":7986,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7933,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11404","reported_seq":18534,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:33.359646+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:33.359646+0000","last_peered":"2024-09-18T11:59:33.359646+0000","last_clean":"2024-09-18T11:59:33.359646+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:33.359646+0000","last_undegraded":"2024-09-18T11:59:33.359646+0000","last_fullsized":"2024-09-18T11:59:33.359646+0000","mapping_epoch":38,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32773330,"num_objects":8003,"num_object_clones":0,"num_object_copies":16006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11402,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11448","reported_seq":11528,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.904136+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:32.904136+0000","last_peered":"2024-09-18T11:59:32.904136+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:32.904136+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32761042,"num_objects":8000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11446,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4468},{"shards":"0,2","objects":1811}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11522","reported_seq":11580,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:33.117216+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:33.117216+0000","last_peered":"2024-09-18T11:59:33.117216+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:33.117216+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32527570,"num_objects":7943,"num_object_clones":0,"num_object_copies":15886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11520,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1762},{"shards":"2","objects":4393}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11511","reported_seq":12327,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:33.159701+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:33.159701+0000","last_peered":"2024-09-18T11:59:33.159701+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:33.159701+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32714752,"num_objects":7987,"num_object_clones":0,"num_object_copies":15974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1466,"num_objects_misplaced":1825,"num_objects_unfound":0,"num_objects_dirty":7987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7991,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1825},{"shards":"2","objects":4696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11512","reported_seq":11596,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.891956+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:32.891956+0000","last_peered":"2024-09-18T11:59:32.891956+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:32.891956+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32834153,"num_objects":8017,"num_object_clones":0,"num_object_copies":16034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11511,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4482},{"shards":"0,2","objects":1788}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11501","reported_seq":18649,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:33.377591+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:33.377591+0000","last_peered":"2024-09-18T11:59:33.377591+0000","last_clean":"2024-09-18T11:59:33.377591+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:33.377591+0000","last_undegraded":"2024-09-18T11:59:33.377591+0000","last_fullsized":"2024-09-18T11:59:33.377591+0000","mapping_epoch":40,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32855867,"num_objects":8024,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11498,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11335","reported_seq":11418,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.786095+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:32.786095+0000","last_peered":"2024-09-18T11:59:32.786095+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:32.786095+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1300","ondisk_log_start":"30'1300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32208082,"num_objects":7865,"num_object_clones":0,"num_object_copies":15730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11333,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4272},{"shards":"0,2","objects":1859}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11453","reported_seq":11540,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:32.929389+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:32.929389+0000","last_peered":"2024-09-18T11:59:32.929389+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:32.929389+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33012132,"num_objects":8063,"num_object_clones":0,"num_object_copies":16126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11450,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4540},{"shards":"0,2","objects":1831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":523788364,"num_objects":127896,"num_object_clones":0,"num_object_copies":255792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33435,"num_objects_misplaced":1825,"num_objects_unfound":0,"num_objects_dirty":127896,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167319,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7330,"num_bytes_recovered":29921897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1147015168,"data_stored":1146937444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160626,"ondisk_log_size":160626,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051652,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667452,"kb_used_data":369580,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93704388,"statfs":{"total":96636764160,"available":95953293312,"internally_reserved":0,"allocated":378449920,"data_stored":378211804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":5.6029999999999998},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.41199999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":5.7750000000000004},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.34100000000000003}]}]},{"osd":1,"up_from":35,"seq":150323855415,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613372,"kb_used_data":311348,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93758468,"statfs":{"total":96636764160,"available":96008671232,"internally_reserved":0,"allocated":318820352,"data_stored":318519471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.20000000000000001},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.433}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.23200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751180,"kb_used_data":440900,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93620660,"statfs":{"total":96636764160,"available":95867555840,"internally_reserved":0,"allocated":451481600,"data_stored":451243934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.308}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.35499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":450662400,"data_stored":450658921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":318418944,"data_stored":318352250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":377933824,"data_stored":377926273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:42.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:42.770 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:42.799 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:42.816 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-18T11:59:41.148007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":515248223,"num_objects":125813,"num_object_clones":0,"num_object_copies":251626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33457,"num_objects_misplaced":1703,"num_objects_unfound":0,"num_objects_dirty":125813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169406,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7431,"num_bytes_recovered":30331516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160814,"ondisk_log_size":160814,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2016440,"kb_used_data":1110428,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281099080,"statfs":{"total":289910292480,"available":287845457920,"internally_reserved":0,"allocated":1137078272,"data_stored":1136301609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,5,7,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17063936,"num_objects":-4166,"num_object_clones":0,"num_object_copies":-8332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":76,"num_objects_misplaced":-261,"num_objects_unfound":0,"num_objects_dirty":-4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4166,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":757760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006932"},"pg_stats":[{"pgid":"2.f","version":"41'11583","reported_seq":11653,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:37.907702+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:37.907702+0000","last_peered":"2024-09-18T11:59:37.907702+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:37.907702+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32575488,"num_objects":7953,"num_object_clones":0,"num_object_copies":15906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11583,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4354},{"shards":"0,2","objects":1784}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11563","reported_seq":11723,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:38.185993+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:38.185993+0000","last_peered":"2024-09-18T11:59:38.185993+0000","last_clean":"2024-09-18T11:59:38.185993+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:38.185993+0000","last_undegraded":"2024-09-18T11:59:38.185993+0000","last_fullsized":"2024-09-18T11:59:38.185993+0000","mapping_epoch":29,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32231424,"num_objects":7869,"num_object_clones":0,"num_object_copies":15738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8755,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11610","reported_seq":11798,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:38.161511+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:38.161511+0000","last_peered":"2024-09-18T11:59:38.161511+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:38.161511+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32550936,"num_objects":7948,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8468,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1706},{"shards":"2","objects":4412}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11516","reported_seq":11547,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:38.176833+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:38.176833+0000","last_peered":"2024-09-18T11:59:38.176833+0000","last_clean":"2024-09-18T11:59:38.176833+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:38.176833+0000","last_undegraded":"2024-09-18T11:59:38.176833+0000","last_fullsized":"2024-09-18T11:59:38.176833+0000","mapping_epoch":29,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":7744,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11516,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11564","reported_seq":11757,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:37.883202+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:37.883202+0000","last_peered":"2024-09-18T11:59:37.883202+0000","last_clean":"2024-09-18T11:59:37.883202+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:37.883202+0000","last_undegraded":"2024-09-18T11:59:37.883202+0000","last_fullsized":"2024-09-18T11:59:37.883202+0000","mapping_epoch":29,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32288768,"num_objects":7883,"num_object_clones":0,"num_object_copies":15766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8729,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11598","reported_seq":11681,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:37.909515+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:37.909515+0000","last_peered":"2024-09-18T11:59:37.909515+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:37.909515+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32052434,"num_objects":7827,"num_object_clones":0,"num_object_copies":15654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11596,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4300},{"shards":"0,2","objects":1643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11520","reported_seq":11609,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:37.816708+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:37.816708+0000","last_peered":"2024-09-18T11:59:37.816708+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:37.816708+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32163026,"num_objects":7854,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4295},{"shards":"0,2","objects":1728}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11730","reported_seq":11957,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:37.831778+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:37.831778+0000","last_peered":"2024-09-18T11:59:37.831778+0000","last_clean":"2024-09-18T11:59:37.831778+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:37.831778+0000","last_undegraded":"2024-09-18T11:59:37.831778+0000","last_fullsized":"2024-09-18T11:59:37.831778+0000","mapping_epoch":29,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32243712,"num_objects":7872,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8047,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11540","reported_seq":18670,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:38.406661+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:38.406661+0000","last_peered":"2024-09-18T11:59:38.406661+0000","last_clean":"2024-09-18T11:59:38.406661+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:38.406661+0000","last_undegraded":"2024-09-18T11:59:38.406661+0000","last_fullsized":"2024-09-18T11:59:38.406661+0000","mapping_epoch":38,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32216274,"num_objects":7867,"num_object_clones":0,"num_object_copies":15734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11538,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11586","reported_seq":11666,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:37.881445+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:37.881445+0000","last_peered":"2024-09-18T11:59:37.881445+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:37.881445+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32195794,"num_objects":7862,"num_object_clones":0,"num_object_copies":15724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11584,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4330},{"shards":"0,2","objects":1673}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11631","reported_seq":11689,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:38.024107+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:38.024107+0000","last_peered":"2024-09-18T11:59:38.024107+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:38.024107+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32081106,"num_objects":7834,"num_object_clones":0,"num_object_copies":15668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11629,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1653},{"shards":"2","objects":4284}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11633","reported_seq":12649,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:38.142989+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:38.142989+0000","last_peered":"2024-09-18T11:59:38.142989+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:38.142989+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32215040,"num_objects":7865,"num_object_clones":0,"num_object_copies":15730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1488,"num_objects_misplaced":1703,"num_objects_unfound":0,"num_objects_dirty":7865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8113,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":1646592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1703},{"shards":"2","objects":4674}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11661","reported_seq":11740,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:37.866869+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:37.866869+0000","last_peered":"2024-09-18T11:59:37.866869+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:37.866869+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32223849,"num_objects":7868,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11660,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4333},{"shards":"0,2","objects":1639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11629","reported_seq":18777,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:38.323179+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:38.323179+0000","last_peered":"2024-09-18T11:59:38.323179+0000","last_clean":"2024-09-18T11:59:38.323179+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:38.323179+0000","last_undegraded":"2024-09-18T11:59:38.323179+0000","last_fullsized":"2024-09-18T11:59:38.323179+0000","mapping_epoch":40,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32331579,"num_objects":7896,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11626,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11458","reported_seq":11540,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:37.874187+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:37.874187+0000","last_peered":"2024-09-18T11:59:37.874187+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:37.874187+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1400","ondisk_log_start":"30'1400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31704274,"num_objects":7742,"num_object_clones":0,"num_object_copies":15484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11456,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4149},{"shards":"0,2","objects":1736}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11589","reported_seq":11676,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:37.902546+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:37.902546+0000","last_peered":"2024-09-18T11:59:37.902546+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:37.902546+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32455076,"num_objects":7927,"num_object_clones":0,"num_object_copies":15854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11586,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4404},{"shards":"0,2","objects":1695}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":515248204,"num_objects":125811,"num_object_clones":0,"num_object_copies":251622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33457,"num_objects_misplaced":1703,"num_objects_unfound":0,"num_objects_dirty":125811,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169404,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7430,"num_bytes_recovered":30331497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1135390720,"data_stored":1135312996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160811,"ondisk_log_size":160811,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051653,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667588,"kb_used_data":365564,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93704252,"statfs":{"total":96636764160,"available":95953154048,"internally_reserved":0,"allocated":374337536,"data_stored":374099420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.20300000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.27700000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":5.3920000000000003},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":5.4089999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855416,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616472,"kb_used_data":310288,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93755368,"statfs":{"total":96636764160,"available":96005496832,"internally_reserved":0,"allocated":317734912,"data_stored":317434031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.21299999999999999},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.376}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.40999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732380,"kb_used_data":434576,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93639460,"statfs":{"total":96636764160,"available":95886807040,"internally_reserved":0,"allocated":445005824,"data_stored":444768158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,0,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.63}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":5.7800000000000002},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":444239872,"data_stored":444236393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":317345792,"data_stored":317279098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":373805056,"data_stored":373797505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:42.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:43.486 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:43.499 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-18T11:59:43.155599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":510529631,"num_objects":124661,"num_object_clones":0,"num_object_copies":249322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33457,"num_objects_misplaced":1703,"num_objects_unfound":0,"num_objects_dirty":124661,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170558,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7431,"num_bytes_recovered":30331516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160566,"ondisk_log_size":160566,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2018636,"kb_used_data":1104308,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281096884,"statfs":{"total":289910292480,"available":287843209216,"internally_reserved":0,"allocated":1130811392,"data_stored":1130034729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,4,10,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21782528,"num_objects":-5318,"num_object_clones":0,"num_object_copies":-10636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":76,"num_objects_misplaced":-261,"num_objects_unfound":0,"num_objects_dirty":-5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":757760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.014009"},"pg_stats":[{"pgid":"2.f","version":"41'11730","reported_seq":11800,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.899737+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:42.899737+0000","last_peered":"2024-09-18T11:59:42.899737+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:42.899737+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31973376,"num_objects":7806,"num_object_clones":0,"num_object_copies":15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11730,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4207},{"shards":"0,2","objects":1637}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11563","reported_seq":11723,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:38.185993+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:38.185993+0000","last_peered":"2024-09-18T11:59:38.185993+0000","last_clean":"2024-09-18T11:59:38.185993+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:38.185993+0000","last_undegraded":"2024-09-18T11:59:38.185993+0000","last_fullsized":"2024-09-18T11:59:38.185993+0000","mapping_epoch":29,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32231424,"num_objects":7869,"num_object_clones":0,"num_object_copies":15738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8755,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11610","reported_seq":11798,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:38.161511+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:38.161511+0000","last_peered":"2024-09-18T11:59:38.161511+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:38.161511+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32550936,"num_objects":7948,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8468,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1706},{"shards":"2","objects":4412}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11516","reported_seq":11547,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:38.176833+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:38.176833+0000","last_peered":"2024-09-18T11:59:38.176833+0000","last_clean":"2024-09-18T11:59:38.176833+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:38.176833+0000","last_undegraded":"2024-09-18T11:59:38.176833+0000","last_fullsized":"2024-09-18T11:59:38.176833+0000","mapping_epoch":29,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":7744,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11516,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11677","reported_seq":11870,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:42.928954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:42.928954+0000","last_peered":"2024-09-18T11:59:42.928954+0000","last_clean":"2024-09-18T11:59:42.928954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:42.928954+0000","last_undegraded":"2024-09-18T11:59:42.928954+0000","last_fullsized":"2024-09-18T11:59:42.928954+0000","mapping_epoch":29,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31825920,"num_objects":7770,"num_object_clones":0,"num_object_copies":15540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8842,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11719","reported_seq":11802,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.802000+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:42.802000+0000","last_peered":"2024-09-18T11:59:42.802000+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:42.802000+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31556818,"num_objects":7706,"num_object_clones":0,"num_object_copies":15412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11717,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4179},{"shards":"0,2","objects":1522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11637","reported_seq":11726,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.906467+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:42.906467+0000","last_peered":"2024-09-18T11:59:42.906467+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:42.906467+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31683794,"num_objects":7737,"num_object_clones":0,"num_object_copies":15474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4178},{"shards":"0,2","objects":1611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11868","reported_seq":12095,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:42.923561+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:42.923561+0000","last_peered":"2024-09-18T11:59:42.923561+0000","last_clean":"2024-09-18T11:59:42.923561+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:42.923561+0000","last_undegraded":"2024-09-18T11:59:42.923561+0000","last_fullsized":"2024-09-18T11:59:42.923561+0000","mapping_epoch":29,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31678464,"num_objects":7734,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8185,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11540","reported_seq":18670,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:38.406661+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:38.406661+0000","last_peered":"2024-09-18T11:59:38.406661+0000","last_clean":"2024-09-18T11:59:38.406661+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:38.406661+0000","last_undegraded":"2024-09-18T11:59:38.406661+0000","last_fullsized":"2024-09-18T11:59:38.406661+0000","mapping_epoch":38,"log_start":"30'1500","ondisk_log_start":"30'1500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32216274,"num_objects":7867,"num_object_clones":0,"num_object_copies":15734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11538,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11706","reported_seq":11786,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.866673+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:42.866673+0000","last_peered":"2024-09-18T11:59:42.866673+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:42.866673+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31704274,"num_objects":7742,"num_object_clones":0,"num_object_copies":15484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11704,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4210},{"shards":"0,2","objects":1553}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11631","reported_seq":11689,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:38.024107+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:38.024107+0000","last_peered":"2024-09-18T11:59:38.024107+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:38.024107+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32081106,"num_objects":7834,"num_object_clones":0,"num_object_copies":15668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11629,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1653},{"shards":"2","objects":4284}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11633","reported_seq":12649,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:38.142989+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:38.142989+0000","last_peered":"2024-09-18T11:59:38.142989+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:38.142989+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32215040,"num_objects":7865,"num_object_clones":0,"num_object_copies":15730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1488,"num_objects_misplaced":1703,"num_objects_unfound":0,"num_objects_dirty":7865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8113,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":1646592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1703},{"shards":"2","objects":4674}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11779","reported_seq":11861,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.933518+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:42.933518+0000","last_peered":"2024-09-18T11:59:42.933518+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:42.933518+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31740521,"num_objects":7750,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11778,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4215},{"shards":"0,2","objects":1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11629","reported_seq":18777,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:38.323179+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:38.323179+0000","last_peered":"2024-09-18T11:59:38.323179+0000","last_clean":"2024-09-18T11:59:38.323179+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:38.323179+0000","last_undegraded":"2024-09-18T11:59:38.323179+0000","last_fullsized":"2024-09-18T11:59:38.323179+0000","mapping_epoch":40,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32331579,"num_objects":7896,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11626,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11604","reported_seq":11687,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.851405+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:42.851405+0000","last_peered":"2024-09-18T11:59:42.851405+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:42.851405+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31106258,"num_objects":7596,"num_object_clones":0,"num_object_copies":15192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11602,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4003},{"shards":"0,2","objects":1590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11721","reported_seq":11808,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.951127+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:42.951127+0000","last_peered":"2024-09-18T11:59:42.951127+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:42.951127+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31914404,"num_objects":7795,"num_object_clones":0,"num_object_copies":15590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11718,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4272},{"shards":"0,2","objects":1563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":510529612,"num_objects":124659,"num_object_clones":0,"num_object_copies":249318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33457,"num_objects_misplaced":1703,"num_objects_unfound":0,"num_objects_dirty":124659,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170556,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7430,"num_bytes_recovered":30331497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1129103360,"data_stored":1129025636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051653,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667588,"kb_used_data":365564,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93704252,"statfs":{"total":96636764160,"available":95953154048,"internally_reserved":0,"allocated":374337536,"data_stored":374099420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.20300000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.27700000000000002}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":5.3920000000000003},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":5.4089999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855416,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616472,"kb_used_data":310288,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93755368,"statfs":{"total":96636764160,"available":96005496832,"internally_reserved":0,"allocated":317734912,"data_stored":317434031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.21299999999999999},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.376}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.40999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734576,"kb_used_data":428456,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93637264,"statfs":{"total":96636764160,"available":95884558336,"internally_reserved":0,"allocated":438738944,"data_stored":438501278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.23899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":3.7789999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":437952512,"data_stored":437949033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":317345792,"data_stored":317279098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":373805056,"data_stored":373797505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:43.501 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:59:43.501 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'11730", 'reported_seq': 11800, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:42.899737+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:59:42.899737+0000', 'last_peered': '2024-09-18T11:59:42.899737+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:59:42.899737+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'1700", 'ondisk_log_start': "30'1700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31973376, 'num_objects': 7806, 'num_object_clones': 0, 'num_object_copies': 15612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11730, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4207}, {'shards': '0,2', 'objects': 1637}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:43.501 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:59:43.501 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'11610", 'reported_seq': 11798, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:38.161511+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:59:38.161511+0000', 'last_peered': '2024-09-18T11:59:38.161511+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:59:38.161511+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'1600", 'ondisk_log_start': "30'1600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32550936, 'num_objects': 7948, 'num_object_clones': 0, 'num_object_copies': 15896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7948, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8468, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1706}, {'shards': '2', 'objects': 4412}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:43.501 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:59:43.501 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'11719", 'reported_seq': 11802, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:42.802000+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:59:42.802000+0000', 'last_peered': '2024-09-18T11:59:42.802000+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:59:42.802000+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'1700", 'ondisk_log_start': "30'1700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31556818, 'num_objects': 7706, 'num_object_clones': 0, 'num_object_copies': 15412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11717, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4179}, {'shards': '0,2', 'objects': 1522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:43.501 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:59:43.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'11637", 'reported_seq': 11726, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:42.906467+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:59:42.906467+0000', 'last_peered': '2024-09-18T11:59:42.906467+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:59:42.906467+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'1600", 'ondisk_log_start': "30'1600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31683794, 'num_objects': 7737, 'num_object_clones': 0, 'num_object_copies': 15474, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7737, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11635, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4178}, {'shards': '0,2', 'objects': 1611}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:43.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:59:43.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'11706", 'reported_seq': 11786, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:42.866673+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:59:42.866673+0000', 'last_peered': '2024-09-18T11:59:42.866673+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:59:42.866673+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'1700", 'ondisk_log_start': "30'1700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31704274, 'num_objects': 7742, 'num_object_clones': 0, 'num_object_copies': 15484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11704, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4210}, {'shards': '0,2', 'objects': 1553}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:43.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:59:43.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'11631", 'reported_seq': 11689, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:38.024107+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:59:38.024107+0000', 'last_peered': '2024-09-18T11:59:38.024107+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:59:38.024107+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'1600", 'ondisk_log_start': "30'1600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32081106, 'num_objects': 7834, 'num_object_clones': 0, 'num_object_copies': 15668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11629, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1653}, {'shards': '2', 'objects': 4284}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:43.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:59:43.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'11633", 'reported_seq': 12649, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:38.142989+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T11:59:38.142989+0000', 'last_peered': '2024-09-18T11:59:38.142989+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:59:38.142989+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'1600", 'ondisk_log_start': "30'1600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32215040, 'num_objects': 7865, 'num_object_clones': 0, 'num_object_copies': 15730, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1488, 'num_objects_misplaced': 1703, 'num_objects_unfound': 0, 'num_objects_dirty': 7865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8113, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 402, 'num_bytes_recovered': 1646592, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1703}, {'shards': '2', 'objects': 4674}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:43.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:59:43.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'11779", 'reported_seq': 11861, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:42.933518+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:59:42.933518+0000', 'last_peered': '2024-09-18T11:59:42.933518+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:59:42.933518+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'1700", 'ondisk_log_start': "30'1700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31740521, 'num_objects': 7750, 'num_object_clones': 0, 'num_object_copies': 15500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11778, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4215}, {'shards': '0,2', 'objects': 1521}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:43.502 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:59:43.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'11604", 'reported_seq': 11687, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:42.851405+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:59:42.851405+0000', 'last_peered': '2024-09-18T11:59:42.851405+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:59:42.851405+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'1600", 'ondisk_log_start': "30'1600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31106258, 'num_objects': 7596, 'num_object_clones': 0, 'num_object_copies': 15192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11602, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4003}, {'shards': '0,2', 'objects': 1590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:43.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:59:43.503 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'11721", 'reported_seq': 11808, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:42.951127+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:59:42.951127+0000', 'last_peered': '2024-09-18T11:59:42.951127+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:59:42.951127+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'1700", 'ondisk_log_start': "30'1700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31914404, 'num_objects': 7795, 'num_object_clones': 0, 'num_object_copies': 15590, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7795, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11718, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4272}, {'shards': '0,2', 'objects': 1563}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:43.503 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:59:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:44.670 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33457/249322 objects degraded (13.419%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":384,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":124661,"data_bytes":510529631,"bytes_used":2067083264,"bytes_avail":287843209216,"bytes_total":289910292480,"degraded_objects":33457,"degraded_total":249322,"degraded_ratio":0.13419192851011943,"misplaced_objects":1703,"misplaced_total":249322,"misplaced_ratio":0.006830524382124321,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":63073,"recovering_keys_per_sec":0,"num_objects_recovered":185,"num_bytes_recovered":757760,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":442},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:59:44.689 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:59:44.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:45.428 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:45.444 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:45.457 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-18T11:59:45.156063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":506593375,"num_objects":123700,"num_object_clones":0,"num_object_copies":247400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33506,"num_objects_misplaced":1566,"num_objects_unfound":0,"num_objects_dirty":123700,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171519,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7521,"num_bytes_recovered":30691964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2021516,"kb_used_data":1098868,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281094004,"statfs":{"total":289910292480,"available":287840260096,"internally_reserved":0,"allocated":1125240832,"data_stored":1124471127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,5,6,7,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20856832,"num_objects":-5092,"num_object_clones":0,"num_object_copies":-10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":125,"num_objects_misplaced":-398,"num_objects_unfound":0,"num_objects_dirty":-5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":1118208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.014040"},"pg_stats":[{"pgid":"2.f","version":"41'11730","reported_seq":11800,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.899737+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:42.899737+0000","last_peered":"2024-09-18T11:59:42.899737+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:42.899737+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31973376,"num_objects":7806,"num_object_clones":0,"num_object_copies":15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11730,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4207},{"shards":"0,2","objects":1637}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11688","reported_seq":11847,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:43.164571+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:43.164571+0000","last_peered":"2024-09-18T11:59:43.164571+0000","last_clean":"2024-09-18T11:59:43.164571+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:43.164571+0000","last_undegraded":"2024-09-18T11:59:43.164571+0000","last_fullsized":"2024-09-18T11:59:43.164571+0000","mapping_epoch":29,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":7744,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8880,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11737","reported_seq":11925,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:43.192245+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:43.192245+0000","last_peered":"2024-09-18T11:59:43.192245+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:43.192245+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32030744,"num_objects":7821,"num_object_clones":0,"num_object_copies":15642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7821,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8595,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1579},{"shards":"2","objects":4285}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11644","reported_seq":11675,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:43.160150+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:43.160150+0000","last_peered":"2024-09-18T11:59:43.160150+0000","last_clean":"2024-09-18T11:59:43.160150+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:43.160150+0000","last_undegraded":"2024-09-18T11:59:43.160150+0000","last_fullsized":"2024-09-18T11:59:43.160150+0000","mapping_epoch":29,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":7616,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11644,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11677","reported_seq":11870,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:42.928954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:42.928954+0000","last_peered":"2024-09-18T11:59:42.928954+0000","last_clean":"2024-09-18T11:59:42.928954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:42.928954+0000","last_undegraded":"2024-09-18T11:59:42.928954+0000","last_fullsized":"2024-09-18T11:59:42.928954+0000","mapping_epoch":29,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31825920,"num_objects":7770,"num_object_clones":0,"num_object_copies":15540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8842,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11719","reported_seq":11802,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.802000+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:42.802000+0000","last_peered":"2024-09-18T11:59:42.802000+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:42.802000+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31556818,"num_objects":7706,"num_object_clones":0,"num_object_copies":15412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11717,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4179},{"shards":"0,2","objects":1522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11637","reported_seq":11726,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.906467+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:42.906467+0000","last_peered":"2024-09-18T11:59:42.906467+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:42.906467+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31683794,"num_objects":7737,"num_object_clones":0,"num_object_copies":15474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4178},{"shards":"0,2","objects":1611}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'11868","reported_seq":12095,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:42.923561+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:42.923561+0000","last_peered":"2024-09-18T11:59:42.923561+0000","last_clean":"2024-09-18T11:59:42.923561+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:42.923561+0000","last_undegraded":"2024-09-18T11:59:42.923561+0000","last_fullsized":"2024-09-18T11:59:42.923561+0000","mapping_epoch":29,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31678464,"num_objects":7734,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8185,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11689","reported_seq":18819,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:43.482194+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:43.482194+0000","last_peered":"2024-09-18T11:59:43.482194+0000","last_clean":"2024-09-18T11:59:43.482194+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:43.482194+0000","last_undegraded":"2024-09-18T11:59:43.482194+0000","last_fullsized":"2024-09-18T11:59:43.482194+0000","mapping_epoch":38,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31605970,"num_objects":7718,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11687,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11706","reported_seq":11786,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.866673+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:42.866673+0000","last_peered":"2024-09-18T11:59:42.866673+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:42.866673+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31704274,"num_objects":7742,"num_object_clones":0,"num_object_copies":15484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11704,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4210},{"shards":"0,2","objects":1553}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11790","reported_seq":11843,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:43.183215+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:43.183215+0000","last_peered":"2024-09-18T11:59:43.183215+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:43.183215+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31429842,"num_objects":7675,"num_object_clones":0,"num_object_copies":15350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1494},{"shards":"2","objects":4125}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11775","reported_seq":12971,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:43.217671+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:43.217671+0000","last_peered":"2024-09-18T11:59:43.217671+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:43.217671+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31633408,"num_objects":7723,"num_object_clones":0,"num_object_copies":15446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1537,"num_objects_misplaced":1566,"num_objects_unfound":0,"num_objects_dirty":7723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8255,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":2007040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1566},{"shards":"2","objects":4620}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11779","reported_seq":11861,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.933518+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:42.933518+0000","last_peered":"2024-09-18T11:59:42.933518+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:42.933518+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31740521,"num_objects":7750,"num_object_clones":0,"num_object_copies":15500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11778,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4215},{"shards":"0,2","objects":1521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11760","reported_seq":18908,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:43.415452+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:43.415452+0000","last_peered":"2024-09-18T11:59:43.415452+0000","last_clean":"2024-09-18T11:59:43.415452+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:43.415452+0000","last_undegraded":"2024-09-18T11:59:43.415452+0000","last_fullsized":"2024-09-18T11:59:43.415452+0000","mapping_epoch":40,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31795003,"num_objects":7765,"num_object_clones":0,"num_object_copies":15530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11604","reported_seq":11687,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.851405+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:42.851405+0000","last_peered":"2024-09-18T11:59:42.851405+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:42.851405+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1600","ondisk_log_start":"30'1600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31106258,"num_objects":7596,"num_object_clones":0,"num_object_copies":15192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11602,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4003},{"shards":"0,2","objects":1590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11721","reported_seq":11808,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:42.951127+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:42.951127+0000","last_peered":"2024-09-18T11:59:42.951127+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:42.951127+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31914404,"num_objects":7795,"num_object_clones":0,"num_object_copies":15590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11718,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4272},{"shards":"0,2","objects":1563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":506593356,"num_objects":123698,"num_object_clones":0,"num_object_copies":247396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33506,"num_objects_misplaced":1566,"num_objects_unfound":0,"num_objects_dirty":123698,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171517,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7520,"num_bytes_recovered":30691945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1123524608,"data_stored":1123453842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160824,"ondisk_log_size":160824,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051654,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667420,"kb_used_data":361236,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93704420,"statfs":{"total":96636764160,"available":95953326080,"internally_reserved":0,"allocated":369905664,"data_stored":369667548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":4.4359999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":4.476},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.33200000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855417,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619520,"kb_used_data":309176,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93752320,"statfs":{"total":96636764160,"available":96002375680,"internally_reserved":0,"allocated":316596224,"data_stored":316302301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.22900000000000001},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":1.341}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.26000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734576,"kb_used_data":428456,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93637264,"statfs":{"total":96636764160,"available":95884558336,"internally_reserved":0,"allocated":438738944,"data_stored":438501278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.23899999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":3.7789999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":437952512,"data_stored":437949033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316186624,"data_stored":316126888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":369385472,"data_stored":369377921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:48.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:49.178 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:49.199 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:49.224 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-18T11:59:49.156879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":497905759,"num_objects":121579,"num_object_clones":0,"num_object_copies":243158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33552,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":121579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173640,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7612,"num_bytes_recovered":31064700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160548,"ondisk_log_size":160548,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2009952,"kb_used_data":1087296,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281105568,"statfs":{"total":289910292480,"available":287852101632,"internally_reserved":0,"allocated":1113391104,"data_stored":1112621399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,5,4,6,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25882624,"num_objects":-6319,"num_object_clones":0,"num_object_copies":-12638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":117,"num_objects_misplaced":-396,"num_objects_unfound":0,"num_objects_dirty":-6319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6319,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":1142784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012154"},"pg_stats":[{"pgid":"2.f","version":"41'11870","reported_seq":11941,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.995401+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:47.995401+0000","last_peered":"2024-09-18T11:59:47.995401+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:47.995401+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31399936,"num_objects":7666,"num_object_clones":0,"num_object_copies":15332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11870,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4067},{"shards":"0,2","objects":1497}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11830","reported_seq":11990,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:48.196359+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:48.196359+0000","last_peered":"2024-09-18T11:59:48.196359+0000","last_clean":"2024-09-18T11:59:48.196359+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:48.196359+0000","last_undegraded":"2024-09-18T11:59:48.196359+0000","last_fullsized":"2024-09-18T11:59:48.196359+0000","mapping_epoch":29,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31137792,"num_objects":7602,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11849","reported_seq":12036,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:48.231532+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:48.231532+0000","last_peered":"2024-09-18T11:59:48.231532+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:48.231532+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31571992,"num_objects":7709,"num_object_clones":0,"num_object_copies":15418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7709,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8707,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1467},{"shards":"2","objects":4173}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11776","reported_seq":11807,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:48.099615+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:48.099615+0000","last_peered":"2024-09-18T11:59:48.099615+0000","last_clean":"2024-09-18T11:59:48.099615+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:48.099615+0000","last_undegraded":"2024-09-18T11:59:48.099615+0000","last_fullsized":"2024-09-18T11:59:48.099615+0000","mapping_epoch":29,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30654464,"num_objects":7484,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11776,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11808","reported_seq":12001,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:47.993436+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:47.993436+0000","last_peered":"2024-09-18T11:59:47.993436+0000","last_clean":"2024-09-18T11:59:47.993436+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:47.993436+0000","last_undegraded":"2024-09-18T11:59:47.993436+0000","last_fullsized":"2024-09-18T11:59:47.993436+0000","mapping_epoch":29,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31289344,"num_objects":7639,"num_object_clones":0,"num_object_copies":15278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8973,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11832","reported_seq":11915,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.954956+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:47.954956+0000","last_peered":"2024-09-18T11:59:47.954956+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:47.954956+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31093970,"num_objects":7593,"num_object_clones":0,"num_object_copies":15186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4066},{"shards":"0,2","objects":1409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11763","reported_seq":11852,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.892725+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:47.892725+0000","last_peered":"2024-09-18T11:59:47.892725+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:47.892725+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31167698,"num_objects":7611,"num_object_clones":0,"num_object_copies":15222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11761,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4052},{"shards":"0,2","objects":1485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12009","reported_seq":12236,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:47.939594+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:47.939594+0000","last_peered":"2024-09-18T11:59:47.939594+0000","last_clean":"2024-09-18T11:59:47.939594+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:47.939594+0000","last_undegraded":"2024-09-18T11:59:47.939594+0000","last_fullsized":"2024-09-18T11:59:47.939594+0000","mapping_epoch":29,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31100928,"num_objects":7593,"num_object_clones":0,"num_object_copies":15186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8326,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11822","reported_seq":18952,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:48.537766+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:48.537766+0000","last_peered":"2024-09-18T11:59:48.537766+0000","last_clean":"2024-09-18T11:59:48.537766+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:48.537766+0000","last_undegraded":"2024-09-18T11:59:48.537766+0000","last_fullsized":"2024-09-18T11:59:48.537766+0000","mapping_epoch":38,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31061202,"num_objects":7585,"num_object_clones":0,"num_object_copies":15170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11820,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11851","reported_seq":11931,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.950193+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:47.950193+0000","last_peered":"2024-09-18T11:59:47.950193+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:47.950193+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31110354,"num_objects":7597,"num_object_clones":0,"num_object_copies":15194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11849,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4065},{"shards":"0,2","objects":1408}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11912","reported_seq":11970,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:48.093877+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:48.093877+0000","last_peered":"2024-09-18T11:59:48.093877+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:48.093877+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30930130,"num_objects":7553,"num_object_clones":0,"num_object_copies":15106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1372},{"shards":"2","objects":4003}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11911","reported_seq":13289,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:48.222482+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:48.222482+0000","last_peered":"2024-09-18T11:59:48.222482+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:48.222482+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31076352,"num_objects":7587,"num_object_clones":0,"num_object_copies":15174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1583,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":7587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":2379776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1429},{"shards":"2","objects":4575}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11905","reported_seq":11989,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.965017+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:47.965017+0000","last_peered":"2024-09-18T11:59:47.965017+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:47.965017+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31224425,"num_objects":7624,"num_object_clones":0,"num_object_copies":15248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11904,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4089},{"shards":"0,2","objects":1395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11912","reported_seq":19060,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:48.509595+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:48.509595+0000","last_peered":"2024-09-18T11:59:48.509595+0000","last_clean":"2024-09-18T11:59:48.509595+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:48.509595+0000","last_undegraded":"2024-09-18T11:59:48.509595+0000","last_fullsized":"2024-09-18T11:59:48.509595+0000","mapping_epoch":40,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31172411,"num_objects":7613,"num_object_clones":0,"num_object_copies":15226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11909,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11745","reported_seq":11828,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.940150+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:47.940150+0000","last_peered":"2024-09-18T11:59:47.940150+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:47.940150+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30528722,"num_objects":7455,"num_object_clones":0,"num_object_copies":14910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11743,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3862},{"shards":"0,2","objects":1449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11850","reported_seq":11937,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.873603+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:47.873603+0000","last_peered":"2024-09-18T11:59:47.873603+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:47.873603+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31386020,"num_objects":7666,"num_object_clones":0,"num_object_copies":15332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11847,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4143},{"shards":"0,2","objects":1434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":497905740,"num_objects":121577,"num_object_clones":0,"num_object_copies":243154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33552,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":121577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173638,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7611,"num_bytes_recovered":31064681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1111437312,"data_stored":1111366546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160545,"ondisk_log_size":160545,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051655,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667360,"kb_used_data":357008,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93704480,"statfs":{"total":96636764160,"available":95953387520,"internally_reserved":0,"allocated":365576192,"data_stored":365338076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":4.4359999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":4.476},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.33200000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855418,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610052,"kb_used_data":308028,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93761788,"statfs":{"total":96636764160,"available":96012070912,"internally_reserved":0,"allocated":315420672,"data_stored":315126749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.20999999999999999},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":4.0720000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":5.694},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.245}]}]},{"osd":0,"up_from":23,"seq":98784247882,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732540,"kb_used_data":422260,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93639300,"statfs":{"total":96636764160,"available":95886643200,"internally_reserved":0,"allocated":432394240,"data_stored":432156574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.161},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.314}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.24099999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.60999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":431304704,"data_stored":431301225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315019264,"data_stored":314959528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365113344,"data_stored":365105793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:49.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:49.983 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:49.993 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:50.016 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-18T11:59:49.156879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":497905759,"num_objects":121579,"num_object_clones":0,"num_object_copies":243158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33552,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":121579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173640,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7612,"num_bytes_recovered":31064700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160548,"ondisk_log_size":160548,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2009952,"kb_used_data":1087296,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281105568,"statfs":{"total":289910292480,"available":287852101632,"internally_reserved":0,"allocated":1113391104,"data_stored":1112621399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,5,4,6,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25882624,"num_objects":-6319,"num_object_clones":0,"num_object_copies":-12638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":117,"num_objects_misplaced":-396,"num_objects_unfound":0,"num_objects_dirty":-6319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6319,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":1142784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012154"},"pg_stats":[{"pgid":"2.f","version":"41'11870","reported_seq":11941,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.995401+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:47.995401+0000","last_peered":"2024-09-18T11:59:47.995401+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:47.995401+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31399936,"num_objects":7666,"num_object_clones":0,"num_object_copies":15332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11870,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4067},{"shards":"0,2","objects":1497}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11830","reported_seq":11990,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:48.196359+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:48.196359+0000","last_peered":"2024-09-18T11:59:48.196359+0000","last_clean":"2024-09-18T11:59:48.196359+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:48.196359+0000","last_undegraded":"2024-09-18T11:59:48.196359+0000","last_fullsized":"2024-09-18T11:59:48.196359+0000","mapping_epoch":29,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31137792,"num_objects":7602,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11849","reported_seq":12036,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:48.231532+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:48.231532+0000","last_peered":"2024-09-18T11:59:48.231532+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:48.231532+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31571992,"num_objects":7709,"num_object_clones":0,"num_object_copies":15418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7709,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8707,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1467},{"shards":"2","objects":4173}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11776","reported_seq":11807,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:48.099615+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:48.099615+0000","last_peered":"2024-09-18T11:59:48.099615+0000","last_clean":"2024-09-18T11:59:48.099615+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:48.099615+0000","last_undegraded":"2024-09-18T11:59:48.099615+0000","last_fullsized":"2024-09-18T11:59:48.099615+0000","mapping_epoch":29,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30654464,"num_objects":7484,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11776,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11808","reported_seq":12001,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:47.993436+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:47.993436+0000","last_peered":"2024-09-18T11:59:47.993436+0000","last_clean":"2024-09-18T11:59:47.993436+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:47.993436+0000","last_undegraded":"2024-09-18T11:59:47.993436+0000","last_fullsized":"2024-09-18T11:59:47.993436+0000","mapping_epoch":29,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31289344,"num_objects":7639,"num_object_clones":0,"num_object_copies":15278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8973,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11832","reported_seq":11915,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.954956+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:47.954956+0000","last_peered":"2024-09-18T11:59:47.954956+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:47.954956+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31093970,"num_objects":7593,"num_object_clones":0,"num_object_copies":15186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4066},{"shards":"0,2","objects":1409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11763","reported_seq":11852,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.892725+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:47.892725+0000","last_peered":"2024-09-18T11:59:47.892725+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:47.892725+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31167698,"num_objects":7611,"num_object_clones":0,"num_object_copies":15222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11761,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4052},{"shards":"0,2","objects":1485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12009","reported_seq":12236,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:47.939594+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:47.939594+0000","last_peered":"2024-09-18T11:59:47.939594+0000","last_clean":"2024-09-18T11:59:47.939594+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:47.939594+0000","last_undegraded":"2024-09-18T11:59:47.939594+0000","last_fullsized":"2024-09-18T11:59:47.939594+0000","mapping_epoch":29,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31100928,"num_objects":7593,"num_object_clones":0,"num_object_copies":15186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8326,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11822","reported_seq":18952,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:48.537766+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:48.537766+0000","last_peered":"2024-09-18T11:59:48.537766+0000","last_clean":"2024-09-18T11:59:48.537766+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:48.537766+0000","last_undegraded":"2024-09-18T11:59:48.537766+0000","last_fullsized":"2024-09-18T11:59:48.537766+0000","mapping_epoch":38,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31061202,"num_objects":7585,"num_object_clones":0,"num_object_copies":15170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11820,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11851","reported_seq":11931,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.950193+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:47.950193+0000","last_peered":"2024-09-18T11:59:47.950193+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:47.950193+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31110354,"num_objects":7597,"num_object_clones":0,"num_object_copies":15194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11849,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4065},{"shards":"0,2","objects":1408}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11912","reported_seq":11970,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:48.093877+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:48.093877+0000","last_peered":"2024-09-18T11:59:48.093877+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:48.093877+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30930130,"num_objects":7553,"num_object_clones":0,"num_object_copies":15106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1372},{"shards":"2","objects":4003}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11911","reported_seq":13289,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:48.222482+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:48.222482+0000","last_peered":"2024-09-18T11:59:48.222482+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:48.222482+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31076352,"num_objects":7587,"num_object_clones":0,"num_object_copies":15174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1583,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":7587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":2379776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1429},{"shards":"2","objects":4575}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11905","reported_seq":11989,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.965017+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:47.965017+0000","last_peered":"2024-09-18T11:59:47.965017+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:47.965017+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31224425,"num_objects":7624,"num_object_clones":0,"num_object_copies":15248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11904,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4089},{"shards":"0,2","objects":1395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11912","reported_seq":19060,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:48.509595+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:48.509595+0000","last_peered":"2024-09-18T11:59:48.509595+0000","last_clean":"2024-09-18T11:59:48.509595+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:48.509595+0000","last_undegraded":"2024-09-18T11:59:48.509595+0000","last_fullsized":"2024-09-18T11:59:48.509595+0000","mapping_epoch":40,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31172411,"num_objects":7613,"num_object_clones":0,"num_object_copies":15226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11909,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11745","reported_seq":11828,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.940150+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:47.940150+0000","last_peered":"2024-09-18T11:59:47.940150+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:47.940150+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30528722,"num_objects":7455,"num_object_clones":0,"num_object_copies":14910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11743,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3862},{"shards":"0,2","objects":1449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11850","reported_seq":11937,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.873603+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:47.873603+0000","last_peered":"2024-09-18T11:59:47.873603+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:47.873603+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31386020,"num_objects":7666,"num_object_clones":0,"num_object_copies":15332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11847,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4143},{"shards":"0,2","objects":1434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":497905740,"num_objects":121577,"num_object_clones":0,"num_object_copies":243154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33552,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":121577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173638,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7611,"num_bytes_recovered":31064681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1111437312,"data_stored":1111366546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160545,"ondisk_log_size":160545,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051655,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667360,"kb_used_data":357008,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93704480,"statfs":{"total":96636764160,"available":95953387520,"internally_reserved":0,"allocated":365576192,"data_stored":365338076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":4.4359999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":4.476},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.33200000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855418,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610052,"kb_used_data":308028,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93761788,"statfs":{"total":96636764160,"available":96012070912,"internally_reserved":0,"allocated":315420672,"data_stored":315126749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.20999999999999999},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":4.0720000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":5.694},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.245}]}]},{"osd":0,"up_from":23,"seq":98784247882,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732540,"kb_used_data":422260,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93639300,"statfs":{"total":96636764160,"available":95886643200,"internally_reserved":0,"allocated":432394240,"data_stored":432156574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.161},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.314}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.24099999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.60999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":431304704,"data_stored":431301225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315019264,"data_stored":314959528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365113344,"data_stored":365105793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:50.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:59:50.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'11870", 'reported_seq': 11941, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:47.995401+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:59:47.995401+0000', 'last_peered': '2024-09-18T11:59:47.995401+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:59:47.995401+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'1800", 'ondisk_log_start': "30'1800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31399936, 'num_objects': 7666, 'num_object_clones': 0, 'num_object_copies': 15332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11870, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4067}, {'shards': '0,2', 'objects': 1497}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:50.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:59:50.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'11849", 'reported_seq': 12036, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:48.231532+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:59:48.231532+0000', 'last_peered': '2024-09-18T11:59:48.231532+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:59:48.231532+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'1800", 'ondisk_log_start': "30'1800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31571992, 'num_objects': 7709, 'num_object_clones': 0, 'num_object_copies': 15418, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7709, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8707, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1467}, {'shards': '2', 'objects': 4173}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:50.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:59:50.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'11832", 'reported_seq': 11915, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:47.954956+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:59:47.954956+0000', 'last_peered': '2024-09-18T11:59:47.954956+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:59:47.954956+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'1800", 'ondisk_log_start': "30'1800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31093970, 'num_objects': 7593, 'num_object_clones': 0, 'num_object_copies': 15186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11830, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4066}, {'shards': '0,2', 'objects': 1409}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:50.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:59:50.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'11763", 'reported_seq': 11852, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:47.892725+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:59:47.892725+0000', 'last_peered': '2024-09-18T11:59:47.892725+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:59:47.892725+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'1700", 'ondisk_log_start': "30'1700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31167698, 'num_objects': 7611, 'num_object_clones': 0, 'num_object_copies': 15222, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7611, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11761, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4052}, {'shards': '0,2', 'objects': 1485}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:50.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:59:50.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'11851", 'reported_seq': 11931, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:47.950193+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:59:47.950193+0000', 'last_peered': '2024-09-18T11:59:47.950193+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:59:47.950193+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'1800", 'ondisk_log_start': "30'1800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31110354, 'num_objects': 7597, 'num_object_clones': 0, 'num_object_copies': 15194, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11849, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4065}, {'shards': '0,2', 'objects': 1408}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:50.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:59:50.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'11912", 'reported_seq': 11970, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:48.093877+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:59:48.093877+0000', 'last_peered': '2024-09-18T11:59:48.093877+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:59:48.093877+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'1900", 'ondisk_log_start': "30'1900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30930130, 'num_objects': 7553, 'num_object_clones': 0, 'num_object_copies': 15106, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1372}, {'shards': '2', 'objects': 4003}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:50.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:59:50.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'11911", 'reported_seq': 13289, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:48.222482+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T11:59:48.222482+0000', 'last_peered': '2024-09-18T11:59:48.222482+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:59:48.222482+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'1900", 'ondisk_log_start': "30'1900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31076352, 'num_objects': 7587, 'num_object_clones': 0, 'num_object_copies': 15174, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1583, 'num_objects_misplaced': 1429, 'num_objects_unfound': 0, 'num_objects_dirty': 7587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8391, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 583, 'num_bytes_recovered': 2379776, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1429}, {'shards': '2', 'objects': 4575}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:50.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:59:50.019 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'11905", 'reported_seq': 11989, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:47.965017+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:59:47.965017+0000', 'last_peered': '2024-09-18T11:59:47.965017+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:59:47.965017+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'1900", 'ondisk_log_start': "30'1900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31224425, 'num_objects': 7624, 'num_object_clones': 0, 'num_object_copies': 15248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11904, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4089}, {'shards': '0,2', 'objects': 1395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:50.019 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:59:50.019 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'11745", 'reported_seq': 11828, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:47.940150+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:59:47.940150+0000', 'last_peered': '2024-09-18T11:59:47.940150+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:59:47.940150+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'1700", 'ondisk_log_start': "30'1700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30528722, 'num_objects': 7455, 'num_object_clones': 0, 'num_object_copies': 14910, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11743, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3862}, {'shards': '0,2', 'objects': 1449}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:50.019 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:59:50.019 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'11850", 'reported_seq': 11937, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:47.873603+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:59:47.873603+0000', 'last_peered': '2024-09-18T11:59:47.873603+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:59:47.873603+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'1800", 'ondisk_log_start': "30'1800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31386020, 'num_objects': 7666, 'num_object_clones': 0, 'num_object_copies': 15332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11847, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4143}, {'shards': '0,2', 'objects': 1434}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:50.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:59:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:50.774 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33552/243158 objects degraded (13.798%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":390,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":121579,"data_bytes":497905759,"bytes_used":2058190848,"bytes_avail":287852101632,"bytes_total":289910292480,"degraded_objects":33552,"degraded_total":243158,"degraded_ratio":0.13798435585092819,"misplaced_objects":1429,"misplaced_total":243158,"misplaced_ratio":0.0058768372827544234,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":95135,"recovering_keys_per_sec":0,"num_objects_recovered":281,"num_bytes_recovered":1142784,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":526},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:59:50.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:59:50.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:51.511 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:51.552 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:51.582 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-18T11:59:51.157357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":497905759,"num_objects":121579,"num_object_clones":0,"num_object_copies":243158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33552,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":121579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173640,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7612,"num_bytes_recovered":31064700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160548,"ondisk_log_size":160548,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2009952,"kb_used_data":1087296,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281105568,"statfs":{"total":289910292480,"available":287852101632,"internally_reserved":0,"allocated":1113391104,"data_stored":1112621399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,5,4,6,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17342464,"num_objects":-4234,"num_object_clones":0,"num_object_copies":-8468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":95,"num_objects_misplaced":-274,"num_objects_unfound":0,"num_objects_dirty":-4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4234,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":733184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009836"},"pg_stats":[{"pgid":"2.f","version":"41'11870","reported_seq":11941,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.995401+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:47.995401+0000","last_peered":"2024-09-18T11:59:47.995401+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:47.995401+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31399936,"num_objects":7666,"num_object_clones":0,"num_object_copies":15332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11870,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4067},{"shards":"0,2","objects":1497}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11830","reported_seq":11990,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:48.196359+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:48.196359+0000","last_peered":"2024-09-18T11:59:48.196359+0000","last_clean":"2024-09-18T11:59:48.196359+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:48.196359+0000","last_undegraded":"2024-09-18T11:59:48.196359+0000","last_fullsized":"2024-09-18T11:59:48.196359+0000","mapping_epoch":29,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31137792,"num_objects":7602,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11849","reported_seq":12036,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:48.231532+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:48.231532+0000","last_peered":"2024-09-18T11:59:48.231532+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:48.231532+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31571992,"num_objects":7709,"num_object_clones":0,"num_object_copies":15418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7709,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8707,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1467},{"shards":"2","objects":4173}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11776","reported_seq":11807,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:48.099615+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:48.099615+0000","last_peered":"2024-09-18T11:59:48.099615+0000","last_clean":"2024-09-18T11:59:48.099615+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:48.099615+0000","last_undegraded":"2024-09-18T11:59:48.099615+0000","last_fullsized":"2024-09-18T11:59:48.099615+0000","mapping_epoch":29,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30654464,"num_objects":7484,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11776,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11808","reported_seq":12001,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:47.993436+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:47.993436+0000","last_peered":"2024-09-18T11:59:47.993436+0000","last_clean":"2024-09-18T11:59:47.993436+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:47.993436+0000","last_undegraded":"2024-09-18T11:59:47.993436+0000","last_fullsized":"2024-09-18T11:59:47.993436+0000","mapping_epoch":29,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31289344,"num_objects":7639,"num_object_clones":0,"num_object_copies":15278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8973,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11832","reported_seq":11915,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.954956+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:47.954956+0000","last_peered":"2024-09-18T11:59:47.954956+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:47.954956+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31093970,"num_objects":7593,"num_object_clones":0,"num_object_copies":15186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4066},{"shards":"0,2","objects":1409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11763","reported_seq":11852,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.892725+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:47.892725+0000","last_peered":"2024-09-18T11:59:47.892725+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:47.892725+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31167698,"num_objects":7611,"num_object_clones":0,"num_object_copies":15222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11761,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4052},{"shards":"0,2","objects":1485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12009","reported_seq":12236,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:47.939594+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:47.939594+0000","last_peered":"2024-09-18T11:59:47.939594+0000","last_clean":"2024-09-18T11:59:47.939594+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:47.939594+0000","last_undegraded":"2024-09-18T11:59:47.939594+0000","last_fullsized":"2024-09-18T11:59:47.939594+0000","mapping_epoch":29,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31100928,"num_objects":7593,"num_object_clones":0,"num_object_copies":15186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8326,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11822","reported_seq":18952,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:48.537766+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:48.537766+0000","last_peered":"2024-09-18T11:59:48.537766+0000","last_clean":"2024-09-18T11:59:48.537766+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:48.537766+0000","last_undegraded":"2024-09-18T11:59:48.537766+0000","last_fullsized":"2024-09-18T11:59:48.537766+0000","mapping_epoch":38,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31061202,"num_objects":7585,"num_object_clones":0,"num_object_copies":15170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11820,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11851","reported_seq":11931,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.950193+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:47.950193+0000","last_peered":"2024-09-18T11:59:47.950193+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:47.950193+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31110354,"num_objects":7597,"num_object_clones":0,"num_object_copies":15194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11849,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4065},{"shards":"0,2","objects":1408}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'11912","reported_seq":11970,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:48.093877+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:48.093877+0000","last_peered":"2024-09-18T11:59:48.093877+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:48.093877+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30930130,"num_objects":7553,"num_object_clones":0,"num_object_copies":15106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1372},{"shards":"2","objects":4003}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'11911","reported_seq":13289,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:48.222482+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:48.222482+0000","last_peered":"2024-09-18T11:59:48.222482+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:48.222482+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31076352,"num_objects":7587,"num_object_clones":0,"num_object_copies":15174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1583,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":7587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":2379776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1429},{"shards":"2","objects":4575}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'11905","reported_seq":11989,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.965017+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:47.965017+0000","last_peered":"2024-09-18T11:59:47.965017+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:47.965017+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31224425,"num_objects":7624,"num_object_clones":0,"num_object_copies":15248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11904,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4089},{"shards":"0,2","objects":1395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'11912","reported_seq":19060,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:48.509595+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:48.509595+0000","last_peered":"2024-09-18T11:59:48.509595+0000","last_clean":"2024-09-18T11:59:48.509595+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:48.509595+0000","last_undegraded":"2024-09-18T11:59:48.509595+0000","last_fullsized":"2024-09-18T11:59:48.509595+0000","mapping_epoch":40,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31172411,"num_objects":7613,"num_object_clones":0,"num_object_copies":15226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11909,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11745","reported_seq":11828,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.940150+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:47.940150+0000","last_peered":"2024-09-18T11:59:47.940150+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:47.940150+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1700","ondisk_log_start":"30'1700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30528722,"num_objects":7455,"num_object_clones":0,"num_object_copies":14910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11743,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3862},{"shards":"0,2","objects":1449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11850","reported_seq":11937,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:47.873603+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:47.873603+0000","last_peered":"2024-09-18T11:59:47.873603+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:47.873603+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31386020,"num_objects":7666,"num_object_clones":0,"num_object_copies":15332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11847,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4143},{"shards":"0,2","objects":1434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":497905740,"num_objects":121577,"num_object_clones":0,"num_object_copies":243154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33552,"num_objects_misplaced":1429,"num_objects_unfound":0,"num_objects_dirty":121577,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173638,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7611,"num_bytes_recovered":31064681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1111437312,"data_stored":1111366546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160545,"ondisk_log_size":160545,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051655,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667360,"kb_used_data":357008,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93704480,"statfs":{"total":96636764160,"available":95953387520,"internally_reserved":0,"allocated":365576192,"data_stored":365338076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":4.4359999999999999}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":4.476},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.33200000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855418,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610052,"kb_used_data":308028,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93761788,"statfs":{"total":96636764160,"available":96012070912,"internally_reserved":0,"allocated":315420672,"data_stored":315126749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.20999999999999999},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":4.0720000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":5.694},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.245}]}]},{"osd":0,"up_from":23,"seq":98784247882,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732540,"kb_used_data":422260,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93639300,"statfs":{"total":96636764160,"available":95886643200,"internally_reserved":0,"allocated":432394240,"data_stored":432156574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,1,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.161},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.314}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.24099999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.60999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":431304704,"data_stored":431301225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315019264,"data_stored":314959528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365113344,"data_stored":365105793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:54.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:55.237 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:55.278 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:55.296 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-18T11:59:55.158232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":489484383,"num_objects":119523,"num_object_clones":0,"num_object_copies":239046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33572,"num_objects_misplaced":1308,"num_objects_unfound":0,"num_objects_dirty":119523,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175696,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7713,"num_bytes_recovered":31478396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160804,"ondisk_log_size":160804,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1978008,"kb_used_data":1076136,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281137512,"statfs":{"total":289910292480,"available":287884812288,"internally_reserved":0,"allocated":1101963264,"data_stored":1101193559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,2,9,10,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21045248,"num_objects":-5138,"num_object_clones":0,"num_object_copies":-10276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":115,"num_objects_misplaced":-395,"num_objects_unfound":0,"num_objects_dirty":-5138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5138,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002632"},"pg_stats":[{"pgid":"2.f","version":"41'12004","reported_seq":12077,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:52.749310+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:52.749310+0000","last_peered":"2024-09-18T11:59:52.749310+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:52.749310+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30851072,"num_objects":7532,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12004,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3933},{"shards":"0,2","objects":1363}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11964","reported_seq":12124,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:53.210709+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:53.210709+0000","last_peered":"2024-09-18T11:59:53.210709+0000","last_clean":"2024-09-18T11:59:53.210709+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:53.210709+0000","last_undegraded":"2024-09-18T11:59:53.210709+0000","last_fullsized":"2024-09-18T11:59:53.210709+0000","mapping_epoch":29,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30588928,"num_objects":7468,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11979","reported_seq":12167,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.109594+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:53.109594+0000","last_peered":"2024-09-18T11:59:53.109594+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:53.109594+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31039512,"num_objects":7579,"num_object_clones":0,"num_object_copies":15158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8837,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1337},{"shards":"2","objects":4043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11900","reported_seq":11931,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:53.129250+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:53.129250+0000","last_peered":"2024-09-18T11:59:53.129250+0000","last_clean":"2024-09-18T11:59:53.129250+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:53.129250+0000","last_undegraded":"2024-09-18T11:59:53.129250+0000","last_fullsized":"2024-09-18T11:59:53.129250+0000","mapping_epoch":29,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":7360,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11900,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11917","reported_seq":12110,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:52.974202+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:52.974202+0000","last_peered":"2024-09-18T11:59:52.974202+0000","last_clean":"2024-09-18T11:59:52.974202+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:52.974202+0000","last_undegraded":"2024-09-18T11:59:52.974202+0000","last_fullsized":"2024-09-18T11:59:52.974202+0000","mapping_epoch":29,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30842880,"num_objects":7530,"num_object_clones":0,"num_object_copies":15060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9082,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11962","reported_seq":12045,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:52.921541+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:52.921541+0000","last_peered":"2024-09-18T11:59:52.921541+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:52.921541+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30561490,"num_objects":7463,"num_object_clones":0,"num_object_copies":14926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11960,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3936},{"shards":"0,2","objects":1279}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11882","reported_seq":11971,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.000702+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:53.000702+0000","last_peered":"2024-09-18T11:59:53.000702+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:53.000702+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30680274,"num_objects":7492,"num_object_clones":0,"num_object_copies":14984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11880,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3933},{"shards":"0,2","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12141","reported_seq":12368,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:52.989667+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:52.989667+0000","last_peered":"2024-09-18T11:59:52.989667+0000","last_clean":"2024-09-18T11:59:52.989667+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:52.989667+0000","last_undegraded":"2024-09-18T11:59:52.989667+0000","last_fullsized":"2024-09-18T11:59:52.989667+0000","mapping_epoch":29,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30560256,"num_objects":7461,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8458,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11949","reported_seq":19079,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:53.556426+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:53.556426+0000","last_peered":"2024-09-18T11:59:53.556426+0000","last_clean":"2024-09-18T11:59:53.556426+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:53.556426+0000","last_undegraded":"2024-09-18T11:59:53.556426+0000","last_fullsized":"2024-09-18T11:59:53.556426+0000","mapping_epoch":38,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30541010,"num_objects":7458,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11947,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11977","reported_seq":12057,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.015779+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:53.015779+0000","last_peered":"2024-09-18T11:59:53.015779+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:53.015779+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30594258,"num_objects":7471,"num_object_clones":0,"num_object_copies":14942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3939},{"shards":"0,2","objects":1282}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12035","reported_seq":12093,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.125913+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:53.125913+0000","last_peered":"2024-09-18T11:59:53.125913+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:53.125913+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30426322,"num_objects":7430,"num_object_clones":0,"num_object_copies":14860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12033,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1249},{"shards":"2","objects":3880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12036","reported_seq":13615,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.237483+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:53.237483+0000","last_peered":"2024-09-18T11:59:53.237483+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:53.237483+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30564352,"num_objects":7462,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1603,"num_objects_misplaced":1308,"num_objects_unfound":0,"num_objects_dirty":7462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8516,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1308},{"shards":"2","objects":4551}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12036","reported_seq":12120,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:52.935810+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:52.935810+0000","last_peered":"2024-09-18T11:59:52.935810+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:52.935810+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30687849,"num_objects":7493,"num_object_clones":0,"num_object_copies":14986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12035,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3958},{"shards":"0,2","objects":1264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12034","reported_seq":19182,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:53.440520+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:53.440520+0000","last_peered":"2024-09-18T11:59:53.440520+0000","last_clean":"2024-09-18T11:59:53.440520+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:53.440520+0000","last_undegraded":"2024-09-18T11:59:53.440520+0000","last_fullsized":"2024-09-18T11:59:53.440520+0000","mapping_epoch":40,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30672699,"num_objects":7491,"num_object_clones":0,"num_object_copies":14982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11903","reported_seq":11982,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:52.993441+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:52.993441+0000","last_peered":"2024-09-18T11:59:52.993441+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:52.993441+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29881554,"num_objects":7297,"num_object_clones":0,"num_object_copies":14594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11901,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3704},{"shards":"0,2","objects":1291}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11982","reported_seq":12069,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.000175+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:53.000175+0000","last_peered":"2024-09-18T11:59:53.000175+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:53.000175+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30845348,"num_objects":7534,"num_object_clones":0,"num_object_copies":15068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11979,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4011},{"shards":"0,2","objects":1302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":489484364,"num_objects":119521,"num_object_clones":0,"num_object_copies":239042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33572,"num_objects_misplaced":1308,"num_objects_unfound":0,"num_objects_dirty":119521,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175694,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7712,"num_bytes_recovered":31478377,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1099882496,"data_stored":1099811730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051656,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650896,"kb_used_data":353008,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93720944,"statfs":{"total":96636764160,"available":95970246656,"internally_reserved":0,"allocated":361480192,"data_stored":361242076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.35399999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.40899999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855419,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613208,"kb_used_data":307024,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93758632,"statfs":{"total":96636764160,"available":96008839168,"internally_reserved":0,"allocated":314392576,"data_stored":314098653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.71899999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":1.131},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.503}]}]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713904,"kb_used_data":416104,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93657936,"statfs":{"total":96636764160,"available":95905726464,"internally_reserved":0,"allocated":426090496,"data_stored":425852830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.8959999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":8.2170000000000005},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":4.9980000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":424947712,"data_stored":424944233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313970688,"data_stored":313910952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360964096,"data_stored":360956545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:55.298 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:56.007 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:56.018 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:56.041 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-18T11:59:55.158232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":489484383,"num_objects":119523,"num_object_clones":0,"num_object_copies":239046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33572,"num_objects_misplaced":1308,"num_objects_unfound":0,"num_objects_dirty":119523,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175696,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7713,"num_bytes_recovered":31478396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160804,"ondisk_log_size":160804,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1978008,"kb_used_data":1076136,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281137512,"statfs":{"total":289910292480,"available":287884812288,"internally_reserved":0,"allocated":1101963264,"data_stored":1101193559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,2,9,10,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21045248,"num_objects":-5138,"num_object_clones":0,"num_object_copies":-10276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":115,"num_objects_misplaced":-395,"num_objects_unfound":0,"num_objects_dirty":-5138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5138,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002632"},"pg_stats":[{"pgid":"2.f","version":"41'12004","reported_seq":12077,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:52.749310+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:52.749310+0000","last_peered":"2024-09-18T11:59:52.749310+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:52.749310+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30851072,"num_objects":7532,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12004,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3933},{"shards":"0,2","objects":1363}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11964","reported_seq":12124,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:53.210709+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:53.210709+0000","last_peered":"2024-09-18T11:59:53.210709+0000","last_clean":"2024-09-18T11:59:53.210709+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:53.210709+0000","last_undegraded":"2024-09-18T11:59:53.210709+0000","last_fullsized":"2024-09-18T11:59:53.210709+0000","mapping_epoch":29,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30588928,"num_objects":7468,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11979","reported_seq":12167,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.109594+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:53.109594+0000","last_peered":"2024-09-18T11:59:53.109594+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:53.109594+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31039512,"num_objects":7579,"num_object_clones":0,"num_object_copies":15158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8837,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1337},{"shards":"2","objects":4043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11900","reported_seq":11931,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:53.129250+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:53.129250+0000","last_peered":"2024-09-18T11:59:53.129250+0000","last_clean":"2024-09-18T11:59:53.129250+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:53.129250+0000","last_undegraded":"2024-09-18T11:59:53.129250+0000","last_fullsized":"2024-09-18T11:59:53.129250+0000","mapping_epoch":29,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":7360,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11900,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11917","reported_seq":12110,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:52.974202+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:52.974202+0000","last_peered":"2024-09-18T11:59:52.974202+0000","last_clean":"2024-09-18T11:59:52.974202+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:52.974202+0000","last_undegraded":"2024-09-18T11:59:52.974202+0000","last_fullsized":"2024-09-18T11:59:52.974202+0000","mapping_epoch":29,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30842880,"num_objects":7530,"num_object_clones":0,"num_object_copies":15060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9082,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11962","reported_seq":12045,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:52.921541+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:52.921541+0000","last_peered":"2024-09-18T11:59:52.921541+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:52.921541+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30561490,"num_objects":7463,"num_object_clones":0,"num_object_copies":14926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11960,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3936},{"shards":"0,2","objects":1279}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11882","reported_seq":11971,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.000702+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:53.000702+0000","last_peered":"2024-09-18T11:59:53.000702+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:53.000702+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30680274,"num_objects":7492,"num_object_clones":0,"num_object_copies":14984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11880,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3933},{"shards":"0,2","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12141","reported_seq":12368,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:52.989667+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:52.989667+0000","last_peered":"2024-09-18T11:59:52.989667+0000","last_clean":"2024-09-18T11:59:52.989667+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:52.989667+0000","last_undegraded":"2024-09-18T11:59:52.989667+0000","last_fullsized":"2024-09-18T11:59:52.989667+0000","mapping_epoch":29,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30560256,"num_objects":7461,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8458,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11949","reported_seq":19079,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:53.556426+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:53.556426+0000","last_peered":"2024-09-18T11:59:53.556426+0000","last_clean":"2024-09-18T11:59:53.556426+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:53.556426+0000","last_undegraded":"2024-09-18T11:59:53.556426+0000","last_fullsized":"2024-09-18T11:59:53.556426+0000","mapping_epoch":38,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30541010,"num_objects":7458,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11947,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11977","reported_seq":12057,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.015779+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:53.015779+0000","last_peered":"2024-09-18T11:59:53.015779+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:53.015779+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30594258,"num_objects":7471,"num_object_clones":0,"num_object_copies":14942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3939},{"shards":"0,2","objects":1282}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12035","reported_seq":12093,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.125913+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:53.125913+0000","last_peered":"2024-09-18T11:59:53.125913+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:53.125913+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30426322,"num_objects":7430,"num_object_clones":0,"num_object_copies":14860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12033,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1249},{"shards":"2","objects":3880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12036","reported_seq":13615,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.237483+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:53.237483+0000","last_peered":"2024-09-18T11:59:53.237483+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:53.237483+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30564352,"num_objects":7462,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1603,"num_objects_misplaced":1308,"num_objects_unfound":0,"num_objects_dirty":7462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8516,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1308},{"shards":"2","objects":4551}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12036","reported_seq":12120,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:52.935810+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:52.935810+0000","last_peered":"2024-09-18T11:59:52.935810+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:52.935810+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30687849,"num_objects":7493,"num_object_clones":0,"num_object_copies":14986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12035,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3958},{"shards":"0,2","objects":1264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12034","reported_seq":19182,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:53.440520+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:53.440520+0000","last_peered":"2024-09-18T11:59:53.440520+0000","last_clean":"2024-09-18T11:59:53.440520+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:53.440520+0000","last_undegraded":"2024-09-18T11:59:53.440520+0000","last_fullsized":"2024-09-18T11:59:53.440520+0000","mapping_epoch":40,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30672699,"num_objects":7491,"num_object_clones":0,"num_object_copies":14982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11903","reported_seq":11982,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:52.993441+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:52.993441+0000","last_peered":"2024-09-18T11:59:52.993441+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:52.993441+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29881554,"num_objects":7297,"num_object_clones":0,"num_object_copies":14594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11901,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3704},{"shards":"0,2","objects":1291}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11982","reported_seq":12069,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.000175+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:53.000175+0000","last_peered":"2024-09-18T11:59:53.000175+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:53.000175+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30845348,"num_objects":7534,"num_object_clones":0,"num_object_copies":15068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11979,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4011},{"shards":"0,2","objects":1302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":489484364,"num_objects":119521,"num_object_clones":0,"num_object_copies":239042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33572,"num_objects_misplaced":1308,"num_objects_unfound":0,"num_objects_dirty":119521,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175694,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7712,"num_bytes_recovered":31478377,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1099882496,"data_stored":1099811730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051656,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650896,"kb_used_data":353008,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93720944,"statfs":{"total":96636764160,"available":95970246656,"internally_reserved":0,"allocated":361480192,"data_stored":361242076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.35399999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.40899999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855419,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613208,"kb_used_data":307024,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93758632,"statfs":{"total":96636764160,"available":96008839168,"internally_reserved":0,"allocated":314392576,"data_stored":314098653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.71899999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":1.131},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.503}]}]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713904,"kb_used_data":416104,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93657936,"statfs":{"total":96636764160,"available":95905726464,"internally_reserved":0,"allocated":426090496,"data_stored":425852830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.8959999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":8.2170000000000005},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":4.9980000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":424947712,"data_stored":424944233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313970688,"data_stored":313910952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360964096,"data_stored":360956545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T11:59:56.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T11:59:56.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'12004", 'reported_seq': 12077, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:52.749310+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:59:52.749310+0000', 'last_peered': '2024-09-18T11:59:52.749310+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:59:52.749310+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'2000", 'ondisk_log_start': "30'2000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30851072, 'num_objects': 7532, 'num_object_clones': 0, 'num_object_copies': 15064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12004, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3933}, {'shards': '0,2', 'objects': 1363}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:56.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T11:59:56.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'11979", 'reported_seq': 12167, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:53.109594+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:59:53.109594+0000', 'last_peered': '2024-09-18T11:59:53.109594+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:59:53.109594+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'1900", 'ondisk_log_start': "30'1900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31039512, 'num_objects': 7579, 'num_object_clones': 0, 'num_object_copies': 15158, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7579, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8837, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1337}, {'shards': '2', 'objects': 4043}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:56.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T11:59:56.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'11962", 'reported_seq': 12045, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:52.921541+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:59:52.921541+0000', 'last_peered': '2024-09-18T11:59:52.921541+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:59:52.921541+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'1900", 'ondisk_log_start': "30'1900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30561490, 'num_objects': 7463, 'num_object_clones': 0, 'num_object_copies': 14926, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7463, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11960, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3936}, {'shards': '0,2', 'objects': 1279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:56.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T11:59:56.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'11882", 'reported_seq': 11971, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:53.000702+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:59:53.000702+0000', 'last_peered': '2024-09-18T11:59:53.000702+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:59:53.000702+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'1800", 'ondisk_log_start': "30'1800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30680274, 'num_objects': 7492, 'num_object_clones': 0, 'num_object_copies': 14984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11880, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3933}, {'shards': '0,2', 'objects': 1366}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:56.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T11:59:56.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'11977", 'reported_seq': 12057, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:53.015779+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:59:53.015779+0000', 'last_peered': '2024-09-18T11:59:53.015779+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:59:53.015779+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'1900", 'ondisk_log_start': "30'1900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30594258, 'num_objects': 7471, 'num_object_clones': 0, 'num_object_copies': 14942, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7471, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11975, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3939}, {'shards': '0,2', 'objects': 1282}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:56.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T11:59:56.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'12035", 'reported_seq': 12093, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:53.125913+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:59:53.125913+0000', 'last_peered': '2024-09-18T11:59:53.125913+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:59:53.125913+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'2000", 'ondisk_log_start': "30'2000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30426322, 'num_objects': 7430, 'num_object_clones': 0, 'num_object_copies': 14860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7430, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12033, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1249}, {'shards': '2', 'objects': 3880}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:56.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T11:59:56.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'12036", 'reported_seq': 13615, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:53.237483+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T11:59:53.237483+0000', 'last_peered': '2024-09-18T11:59:53.237483+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:59:53.237483+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'2000", 'ondisk_log_start': "30'2000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30564352, 'num_objects': 7462, 'num_object_clones': 0, 'num_object_copies': 14924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1603, 'num_objects_misplaced': 1308, 'num_objects_unfound': 0, 'num_objects_dirty': 7462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8516, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 684, 'num_bytes_recovered': 2793472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1308}, {'shards': '2', 'objects': 4551}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T11:59:56.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T11:59:56.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'12036", 'reported_seq': 12120, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:52.935810+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:59:52.935810+0000', 'last_peered': '2024-09-18T11:59:52.935810+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:59:52.935810+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'2000", 'ondisk_log_start': "30'2000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30687849, 'num_objects': 7493, 'num_object_clones': 0, 'num_object_copies': 14986, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12035, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3958}, {'shards': '0,2', 'objects': 1264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:56.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T11:59:56.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'11903", 'reported_seq': 11982, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:52.993441+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:59:52.993441+0000', 'last_peered': '2024-09-18T11:59:52.993441+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:59:52.993441+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'1900", 'ondisk_log_start': "30'1900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29881554, 'num_objects': 7297, 'num_object_clones': 0, 'num_object_copies': 14594, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7297, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11901, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3704}, {'shards': '0,2', 'objects': 1291}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:56.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T11:59:56.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'11982", 'reported_seq': 12069, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:53.000175+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:59:53.000175+0000', 'last_peered': '2024-09-18T11:59:53.000175+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:59:53.000175+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'1900", 'ondisk_log_start': "30'1900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30845348, 'num_objects': 7534, 'num_object_clones': 0, 'num_object_copies': 15068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7534, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11979, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4011}, {'shards': '0,2', 'objects': 1302}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T11:59:56.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T11:59:56.844 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:56.845 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33572/239046 objects degraded (14.044%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":396,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":119523,"data_bytes":489484383,"bytes_used":2025480192,"bytes_avail":287884812288,"bytes_total":289910292480,"degraded_objects":33572,"degraded_total":239046,"degraded_ratio":0.14044158864820996,"misplaced_objects":1308,"misplaced_total":239046,"misplaced_ratio":0.0054717502070731165,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":95552,"recovering_keys_per_sec":0,"num_objects_recovered":282,"num_bytes_recovered":1146880,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":428},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T11:59:56.875 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T11:59:56.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T11:59:57.677 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T11:59:57.727 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T11:59:57.744 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-18T11:59:57.158694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":489484383,"num_objects":119523,"num_object_clones":0,"num_object_copies":239046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33572,"num_objects_misplaced":1308,"num_objects_unfound":0,"num_objects_dirty":119523,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175696,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7713,"num_bytes_recovered":31478396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160804,"ondisk_log_size":160804,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1978008,"kb_used_data":1076136,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281137512,"statfs":{"total":289910292480,"available":287884812288,"internally_reserved":0,"allocated":1101963264,"data_stored":1101193559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,2,9,10,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17108992,"num_objects":-4177,"num_object_clones":0,"num_object_copies":-8354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66,"num_objects_misplaced":-258,"num_objects_unfound":0,"num_objects_dirty":-4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4177,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002631"},"pg_stats":[{"pgid":"2.f","version":"41'12004","reported_seq":12077,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:52.749310+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:52.749310+0000","last_peered":"2024-09-18T11:59:52.749310+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:52.749310+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30851072,"num_objects":7532,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12004,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3933},{"shards":"0,2","objects":1363}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'11964","reported_seq":12124,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:53.210709+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:53.210709+0000","last_peered":"2024-09-18T11:59:53.210709+0000","last_clean":"2024-09-18T11:59:53.210709+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:53.210709+0000","last_undegraded":"2024-09-18T11:59:53.210709+0000","last_fullsized":"2024-09-18T11:59:53.210709+0000","mapping_epoch":29,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30588928,"num_objects":7468,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'11979","reported_seq":12167,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.109594+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:53.109594+0000","last_peered":"2024-09-18T11:59:53.109594+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:53.109594+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31039512,"num_objects":7579,"num_object_clones":0,"num_object_copies":15158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8837,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1337},{"shards":"2","objects":4043}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'11900","reported_seq":11931,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:53.129250+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:53.129250+0000","last_peered":"2024-09-18T11:59:53.129250+0000","last_clean":"2024-09-18T11:59:53.129250+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:53.129250+0000","last_undegraded":"2024-09-18T11:59:53.129250+0000","last_fullsized":"2024-09-18T11:59:53.129250+0000","mapping_epoch":29,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":7360,"num_object_clones":0,"num_object_copies":14720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11900,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'11917","reported_seq":12110,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:52.974202+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:52.974202+0000","last_peered":"2024-09-18T11:59:52.974202+0000","last_clean":"2024-09-18T11:59:52.974202+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:52.974202+0000","last_undegraded":"2024-09-18T11:59:52.974202+0000","last_fullsized":"2024-09-18T11:59:52.974202+0000","mapping_epoch":29,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30842880,"num_objects":7530,"num_object_clones":0,"num_object_copies":15060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9082,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'11962","reported_seq":12045,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:52.921541+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:52.921541+0000","last_peered":"2024-09-18T11:59:52.921541+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:52.921541+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30561490,"num_objects":7463,"num_object_clones":0,"num_object_copies":14926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11960,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3936},{"shards":"0,2","objects":1279}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'11882","reported_seq":11971,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.000702+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:53.000702+0000","last_peered":"2024-09-18T11:59:53.000702+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:53.000702+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'1800","ondisk_log_start":"30'1800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30680274,"num_objects":7492,"num_object_clones":0,"num_object_copies":14984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11880,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3933},{"shards":"0,2","objects":1366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12141","reported_seq":12368,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:52.989667+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:52.989667+0000","last_peered":"2024-09-18T11:59:52.989667+0000","last_clean":"2024-09-18T11:59:52.989667+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:52.989667+0000","last_undegraded":"2024-09-18T11:59:52.989667+0000","last_fullsized":"2024-09-18T11:59:52.989667+0000","mapping_epoch":29,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30560256,"num_objects":7461,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8458,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'11949","reported_seq":19079,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:53.556426+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:53.556426+0000","last_peered":"2024-09-18T11:59:53.556426+0000","last_clean":"2024-09-18T11:59:53.556426+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:53.556426+0000","last_undegraded":"2024-09-18T11:59:53.556426+0000","last_fullsized":"2024-09-18T11:59:53.556426+0000","mapping_epoch":38,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30541010,"num_objects":7458,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11947,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'11977","reported_seq":12057,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.015779+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:53.015779+0000","last_peered":"2024-09-18T11:59:53.015779+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:53.015779+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30594258,"num_objects":7471,"num_object_clones":0,"num_object_copies":14942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3939},{"shards":"0,2","objects":1282}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12035","reported_seq":12093,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.125913+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:53.125913+0000","last_peered":"2024-09-18T11:59:53.125913+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:53.125913+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30426322,"num_objects":7430,"num_object_clones":0,"num_object_copies":14860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12033,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1249},{"shards":"2","objects":3880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12036","reported_seq":13615,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.237483+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:53.237483+0000","last_peered":"2024-09-18T11:59:53.237483+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:53.237483+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30564352,"num_objects":7462,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1603,"num_objects_misplaced":1308,"num_objects_unfound":0,"num_objects_dirty":7462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8516,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1308},{"shards":"2","objects":4551}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12036","reported_seq":12120,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:52.935810+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:52.935810+0000","last_peered":"2024-09-18T11:59:52.935810+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:52.935810+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30687849,"num_objects":7493,"num_object_clones":0,"num_object_copies":14986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12035,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3958},{"shards":"0,2","objects":1264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12034","reported_seq":19182,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:53.440520+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:53.440520+0000","last_peered":"2024-09-18T11:59:53.440520+0000","last_clean":"2024-09-18T11:59:53.440520+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:53.440520+0000","last_undegraded":"2024-09-18T11:59:53.440520+0000","last_fullsized":"2024-09-18T11:59:53.440520+0000","mapping_epoch":40,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30672699,"num_objects":7491,"num_object_clones":0,"num_object_copies":14982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'11903","reported_seq":11982,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:52.993441+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:52.993441+0000","last_peered":"2024-09-18T11:59:52.993441+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:52.993441+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29881554,"num_objects":7297,"num_object_clones":0,"num_object_copies":14594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11901,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3704},{"shards":"0,2","objects":1291}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'11982","reported_seq":12069,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:53.000175+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:53.000175+0000","last_peered":"2024-09-18T11:59:53.000175+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:53.000175+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'1900","ondisk_log_start":"30'1900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30845348,"num_objects":7534,"num_object_clones":0,"num_object_copies":15068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11979,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4011},{"shards":"0,2","objects":1302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":489484364,"num_objects":119521,"num_object_clones":0,"num_object_copies":239042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33572,"num_objects_misplaced":1308,"num_objects_unfound":0,"num_objects_dirty":119521,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175694,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7712,"num_bytes_recovered":31478377,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1099882496,"data_stored":1099811730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051656,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650896,"kb_used_data":353008,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93720944,"statfs":{"total":96636764160,"available":95970246656,"internally_reserved":0,"allocated":361480192,"data_stored":361242076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.35399999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.40899999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855419,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613208,"kb_used_data":307024,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93758632,"statfs":{"total":96636764160,"available":96008839168,"internally_reserved":0,"allocated":314392576,"data_stored":314098653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.71899999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":1.131},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.503}]}]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713904,"kb_used_data":416104,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93657936,"statfs":{"total":96636764160,"available":95905726464,"internally_reserved":0,"allocated":426090496,"data_stored":425852830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.8959999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":8.2170000000000005},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":4.9980000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":424947712,"data_stored":424944233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313970688,"data_stored":313910952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360964096,"data_stored":360956545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:00.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:01.399 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:01.403 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:01.437 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-09-18T12:00:01.159646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":481042527,"num_objects":117462,"num_object_clones":0,"num_object_copies":234924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33606,"num_objects_misplaced":1175,"num_objects_unfound":0,"num_objects_dirty":117462,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177757,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7812,"num_bytes_recovered":31883900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160765,"ondisk_log_size":160765,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1983116,"kb_used_data":1064620,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281132404,"statfs":{"total":289910292480,"available":287879581696,"internally_reserved":0,"allocated":1090170880,"data_stored":1089401175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,1,6,8,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16863232,"num_objects":-4117,"num_object_clones":0,"num_object_copies":-8234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":-254,"num_objects_unfound":0,"num_objects_dirty":-4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4117,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002767"},"pg_stats":[{"pgid":"2.f","version":"41'12142","reported_seq":12215,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:57.917521+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:57.917521+0000","last_peered":"2024-09-18T11:59:57.917521+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:57.917521+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30285824,"num_objects":7394,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12142,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3795},{"shards":"0,2","objects":1225}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12118","reported_seq":12278,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.228663+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:58.228663+0000","last_peered":"2024-09-18T11:59:58.228663+0000","last_clean":"2024-09-18T11:59:58.228663+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:58.228663+0000","last_undegraded":"2024-09-18T11:59:58.228663+0000","last_fullsized":"2024-09-18T11:59:58.228663+0000","mapping_epoch":29,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29958144,"num_objects":7314,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12112","reported_seq":12300,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.201693+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:58.201693+0000","last_peered":"2024-09-18T11:59:58.201693+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:58.201693+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30494744,"num_objects":7446,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7446,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8970,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1204},{"shards":"2","objects":3910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12015","reported_seq":12046,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.182285+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:58.182285+0000","last_peered":"2024-09-18T11:59:58.182285+0000","last_clean":"2024-09-18T11:59:58.182285+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:58.182285+0000","last_undegraded":"2024-09-18T11:59:58.182285+0000","last_fullsized":"2024-09-18T11:59:58.182285+0000","mapping_epoch":29,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29675520,"num_objects":7245,"num_object_clones":0,"num_object_copies":14490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12015,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12046","reported_seq":12239,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.064545+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:58.064545+0000","last_peered":"2024-09-18T11:59:58.064545+0000","last_clean":"2024-09-18T11:59:58.064545+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:58.064545+0000","last_undegraded":"2024-09-18T11:59:58.064545+0000","last_fullsized":"2024-09-18T11:59:58.064545+0000","mapping_epoch":29,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30314496,"num_objects":7401,"num_object_clones":0,"num_object_copies":14802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9211,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12084","reported_seq":12167,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.047286+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:58.047286+0000","last_peered":"2024-09-18T11:59:58.047286+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:58.047286+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30061778,"num_objects":7341,"num_object_clones":0,"num_object_copies":14682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12082,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3814},{"shards":"0,2","objects":1157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12026","reported_seq":12114,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.009209+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:58.009209+0000","last_peered":"2024-09-18T11:59:58.009209+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:58.009209+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30090450,"num_objects":7348,"num_object_clones":0,"num_object_copies":14696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12024,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3789},{"shards":"0,2","objects":1222}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12276","reported_seq":12503,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.011907+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:58.011907+0000","last_peered":"2024-09-18T11:59:58.011907+0000","last_clean":"2024-09-18T11:59:58.011907+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:58.011907+0000","last_undegraded":"2024-09-18T11:59:58.011907+0000","last_fullsized":"2024-09-18T11:59:58.011907+0000","mapping_epoch":29,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30007296,"num_objects":7326,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8593,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12076","reported_seq":19206,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.570038+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:58.570038+0000","last_peered":"2024-09-18T11:59:58.570038+0000","last_clean":"2024-09-18T11:59:58.570038+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:58.570038+0000","last_undegraded":"2024-09-18T11:59:58.570038+0000","last_fullsized":"2024-09-18T11:59:58.570038+0000","mapping_epoch":38,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30020818,"num_objects":7331,"num_object_clones":0,"num_object_copies":14662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12074,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12118","reported_seq":12198,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.046260+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:58.046260+0000","last_peered":"2024-09-18T11:59:58.046260+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:58.046260+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30016722,"num_objects":7330,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12116,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3798},{"shards":"0,2","objects":1141}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12149","reported_seq":12207,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.198148+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:58.198148+0000","last_peered":"2024-09-18T11:59:58.198148+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:58.198148+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29959378,"num_objects":7316,"num_object_clones":0,"num_object_copies":14632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12147,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1135},{"shards":"2","objects":3766}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12169","reported_seq":13945,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.246563+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:58.246563+0000","last_peered":"2024-09-18T11:59:58.246563+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:58.246563+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30019584,"num_objects":7329,"num_object_clones":0,"num_object_copies":14658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1637,"num_objects_misplaced":1175,"num_objects_unfound":0,"num_objects_dirty":7329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8649,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":3198976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1175},{"shards":"2","objects":4517}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12144","reported_seq":12228,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.018653+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:58.018653+0000","last_peered":"2024-09-18T11:59:58.018653+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:58.018653+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30245481,"num_objects":7385,"num_object_clones":0,"num_object_copies":14770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12143,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3850},{"shards":"0,2","objects":1156}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12159","reported_seq":19307,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.548225+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:58.548225+0000","last_peered":"2024-09-18T11:59:58.548225+0000","last_clean":"2024-09-18T11:59:58.548225+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:58.548225+0000","last_undegraded":"2024-09-18T11:59:58.548225+0000","last_fullsized":"2024-09-18T11:59:58.548225+0000","mapping_epoch":40,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30160699,"num_objects":7366,"num_object_clones":0,"num_object_copies":14732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12156,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12030","reported_seq":12113,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:57.950594+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:57.950594+0000","last_peered":"2024-09-18T11:59:57.950594+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:57.950594+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29361362,"num_objects":7170,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12028,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3577},{"shards":"0,2","objects":1164}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12098","reported_seq":12185,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:57.989218+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:57.989218+0000","last_peered":"2024-09-18T11:59:57.989218+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:57.989218+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30370212,"num_objects":7418,"num_object_clones":0,"num_object_copies":14836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12095,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3895},{"shards":"0,2","objects":1186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":481042508,"num_objects":117460,"num_object_clones":0,"num_object_copies":234920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33606,"num_objects_misplaced":1175,"num_objects_unfound":0,"num_objects_dirty":117460,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177755,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7811,"num_bytes_recovered":31883881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1088241664,"data_stored":1088170898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160762,"ondisk_log_size":160762,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051657,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650828,"kb_used_data":348804,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93721012,"statfs":{"total":96636764160,"available":95970316288,"internally_reserved":0,"allocated":357175296,"data_stored":356937180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.35399999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.40899999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855420,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616360,"kb_used_data":306012,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93755480,"statfs":{"total":96636764160,"available":96005611520,"internally_reserved":0,"allocated":313356288,"data_stored":313062365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.253},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":3.3359999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":4.9210000000000003},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.28399999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715928,"kb_used_data":409804,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93655912,"statfs":{"total":96636764160,"available":95903653888,"internally_reserved":0,"allocated":419639296,"data_stored":419401630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.35399999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.34399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":418582528,"data_stored":418579049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":312942592,"data_stored":312882856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356716544,"data_stored":356708993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:01.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:02.163 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:02.174 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:02.183 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-09-18T12:00:01.159646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":481042527,"num_objects":117462,"num_object_clones":0,"num_object_copies":234924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33606,"num_objects_misplaced":1175,"num_objects_unfound":0,"num_objects_dirty":117462,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177757,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7812,"num_bytes_recovered":31883900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160765,"ondisk_log_size":160765,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1983116,"kb_used_data":1064620,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281132404,"statfs":{"total":289910292480,"available":287879581696,"internally_reserved":0,"allocated":1090170880,"data_stored":1089401175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,1,6,8,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16863232,"num_objects":-4117,"num_object_clones":0,"num_object_copies":-8234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":-254,"num_objects_unfound":0,"num_objects_dirty":-4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4117,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002767"},"pg_stats":[{"pgid":"2.f","version":"41'12142","reported_seq":12215,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:57.917521+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T11:59:57.917521+0000","last_peered":"2024-09-18T11:59:57.917521+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T11:59:57.917521+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30285824,"num_objects":7394,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12142,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3795},{"shards":"0,2","objects":1225}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12118","reported_seq":12278,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.228663+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:58.228663+0000","last_peered":"2024-09-18T11:59:58.228663+0000","last_clean":"2024-09-18T11:59:58.228663+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:58.228663+0000","last_undegraded":"2024-09-18T11:59:58.228663+0000","last_fullsized":"2024-09-18T11:59:58.228663+0000","mapping_epoch":29,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29958144,"num_objects":7314,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12112","reported_seq":12300,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.201693+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:58.201693+0000","last_peered":"2024-09-18T11:59:58.201693+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:58.201693+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30494744,"num_objects":7446,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7446,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8970,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1204},{"shards":"2","objects":3910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12015","reported_seq":12046,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.182285+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:58.182285+0000","last_peered":"2024-09-18T11:59:58.182285+0000","last_clean":"2024-09-18T11:59:58.182285+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:58.182285+0000","last_undegraded":"2024-09-18T11:59:58.182285+0000","last_fullsized":"2024-09-18T11:59:58.182285+0000","mapping_epoch":29,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29675520,"num_objects":7245,"num_object_clones":0,"num_object_copies":14490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12015,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12046","reported_seq":12239,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.064545+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T11:59:58.064545+0000","last_peered":"2024-09-18T11:59:58.064545+0000","last_clean":"2024-09-18T11:59:58.064545+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T11:59:58.064545+0000","last_undegraded":"2024-09-18T11:59:58.064545+0000","last_fullsized":"2024-09-18T11:59:58.064545+0000","mapping_epoch":29,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30314496,"num_objects":7401,"num_object_clones":0,"num_object_copies":14802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9211,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12084","reported_seq":12167,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.047286+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T11:59:58.047286+0000","last_peered":"2024-09-18T11:59:58.047286+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T11:59:58.047286+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30061778,"num_objects":7341,"num_object_clones":0,"num_object_copies":14682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12082,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3814},{"shards":"0,2","objects":1157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12026","reported_seq":12114,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.009209+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T11:59:58.009209+0000","last_peered":"2024-09-18T11:59:58.009209+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T11:59:58.009209+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30090450,"num_objects":7348,"num_object_clones":0,"num_object_copies":14696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12024,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3789},{"shards":"0,2","objects":1222}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12276","reported_seq":12503,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.011907+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T11:59:58.011907+0000","last_peered":"2024-09-18T11:59:58.011907+0000","last_clean":"2024-09-18T11:59:58.011907+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T11:59:58.011907+0000","last_undegraded":"2024-09-18T11:59:58.011907+0000","last_fullsized":"2024-09-18T11:59:58.011907+0000","mapping_epoch":29,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30007296,"num_objects":7326,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8593,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12076","reported_seq":19206,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.570038+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:58.570038+0000","last_peered":"2024-09-18T11:59:58.570038+0000","last_clean":"2024-09-18T11:59:58.570038+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:58.570038+0000","last_undegraded":"2024-09-18T11:59:58.570038+0000","last_fullsized":"2024-09-18T11:59:58.570038+0000","mapping_epoch":38,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30020818,"num_objects":7331,"num_object_clones":0,"num_object_copies":14662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12074,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12118","reported_seq":12198,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.046260+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T11:59:58.046260+0000","last_peered":"2024-09-18T11:59:58.046260+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T11:59:58.046260+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30016722,"num_objects":7330,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12116,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3798},{"shards":"0,2","objects":1141}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12149","reported_seq":12207,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.198148+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:58.198148+0000","last_peered":"2024-09-18T11:59:58.198148+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:58.198148+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29959378,"num_objects":7316,"num_object_clones":0,"num_object_copies":14632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12147,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1135},{"shards":"2","objects":3766}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12169","reported_seq":13945,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.246563+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:58.246563+0000","last_peered":"2024-09-18T11:59:58.246563+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:58.246563+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30019584,"num_objects":7329,"num_object_clones":0,"num_object_copies":14658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1637,"num_objects_misplaced":1175,"num_objects_unfound":0,"num_objects_dirty":7329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8649,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":3198976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1175},{"shards":"2","objects":4517}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12144","reported_seq":12228,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.018653+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T11:59:58.018653+0000","last_peered":"2024-09-18T11:59:58.018653+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T11:59:58.018653+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30245481,"num_objects":7385,"num_object_clones":0,"num_object_copies":14770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12143,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3850},{"shards":"0,2","objects":1156}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12159","reported_seq":19307,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.548225+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:58.548225+0000","last_peered":"2024-09-18T11:59:58.548225+0000","last_clean":"2024-09-18T11:59:58.548225+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:58.548225+0000","last_undegraded":"2024-09-18T11:59:58.548225+0000","last_fullsized":"2024-09-18T11:59:58.548225+0000","mapping_epoch":40,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30160699,"num_objects":7366,"num_object_clones":0,"num_object_copies":14732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12156,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12030","reported_seq":12113,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:57.950594+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T11:59:57.950594+0000","last_peered":"2024-09-18T11:59:57.950594+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T11:59:57.950594+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29361362,"num_objects":7170,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12028,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3577},{"shards":"0,2","objects":1164}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12098","reported_seq":12185,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:57.989218+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T11:59:57.989218+0000","last_peered":"2024-09-18T11:59:57.989218+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T11:59:57.989218+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30370212,"num_objects":7418,"num_object_clones":0,"num_object_copies":14836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12095,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3895},{"shards":"0,2","objects":1186}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":481042508,"num_objects":117460,"num_object_clones":0,"num_object_copies":234920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33606,"num_objects_misplaced":1175,"num_objects_unfound":0,"num_objects_dirty":117460,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177755,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7811,"num_bytes_recovered":31883881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1088241664,"data_stored":1088170898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160762,"ondisk_log_size":160762,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051657,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650828,"kb_used_data":348804,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93721012,"statfs":{"total":96636764160,"available":95970316288,"internally_reserved":0,"allocated":357175296,"data_stored":356937180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.35399999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.40899999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855420,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616360,"kb_used_data":306012,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93755480,"statfs":{"total":96636764160,"available":96005611520,"internally_reserved":0,"allocated":313356288,"data_stored":313062365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.253},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":3.3359999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":4.9210000000000003},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.28399999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715928,"kb_used_data":409804,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93655912,"statfs":{"total":96636764160,"available":95903653888,"internally_reserved":0,"allocated":419639296,"data_stored":419401630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.35399999999999998}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.34399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":418582528,"data_stored":418579049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":312942592,"data_stored":312882856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356716544,"data_stored":356708993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:02.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:00:02.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'12142", 'reported_seq': 12215, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:57.917521+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T11:59:57.917521+0000', 'last_peered': '2024-09-18T11:59:57.917521+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T11:59:57.917521+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'2100", 'ondisk_log_start': "30'2100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30285824, 'num_objects': 7394, 'num_object_clones': 0, 'num_object_copies': 14788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12142, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3795}, {'shards': '0,2', 'objects': 1225}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:02.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:00:02.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'12112", 'reported_seq': 12300, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:58.201693+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T11:59:58.201693+0000', 'last_peered': '2024-09-18T11:59:58.201693+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T11:59:58.201693+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'2100", 'ondisk_log_start': "30'2100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30494744, 'num_objects': 7446, 'num_object_clones': 0, 'num_object_copies': 14892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7446, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8970, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1204}, {'shards': '2', 'objects': 3910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:02.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:00:02.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'12084", 'reported_seq': 12167, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:58.047286+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T11:59:58.047286+0000', 'last_peered': '2024-09-18T11:59:58.047286+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T11:59:58.047286+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'2000", 'ondisk_log_start': "30'2000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30061778, 'num_objects': 7341, 'num_object_clones': 0, 'num_object_copies': 14682, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12082, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3814}, {'shards': '0,2', 'objects': 1157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:02.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:00:02.185 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'12026", 'reported_seq': 12114, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:58.009209+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T11:59:58.009209+0000', 'last_peered': '2024-09-18T11:59:58.009209+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T11:59:58.009209+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'2000", 'ondisk_log_start': "30'2000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30090450, 'num_objects': 7348, 'num_object_clones': 0, 'num_object_copies': 14696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12024, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3789}, {'shards': '0,2', 'objects': 1222}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:02.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:00:02.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'12118", 'reported_seq': 12198, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:58.046260+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T11:59:58.046260+0000', 'last_peered': '2024-09-18T11:59:58.046260+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T11:59:58.046260+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'2100", 'ondisk_log_start': "30'2100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30016722, 'num_objects': 7330, 'num_object_clones': 0, 'num_object_copies': 14660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12116, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3798}, {'shards': '0,2', 'objects': 1141}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:02.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:00:02.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'12149", 'reported_seq': 12207, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:58.198148+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T11:59:58.198148+0000', 'last_peered': '2024-09-18T11:59:58.198148+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T11:59:58.198148+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'2100", 'ondisk_log_start': "30'2100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29959378, 'num_objects': 7316, 'num_object_clones': 0, 'num_object_copies': 14632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12147, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1135}, {'shards': '2', 'objects': 3766}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:02.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:00:02.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'12169", 'reported_seq': 13945, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:58.246563+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T11:59:58.246563+0000', 'last_peered': '2024-09-18T11:59:58.246563+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T11:59:58.246563+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'2100", 'ondisk_log_start': "30'2100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30019584, 'num_objects': 7329, 'num_object_clones': 0, 'num_object_copies': 14658, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1637, 'num_objects_misplaced': 1175, 'num_objects_unfound': 0, 'num_objects_dirty': 7329, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8649, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 783, 'num_bytes_recovered': 3198976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1175}, {'shards': '2', 'objects': 4517}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:02.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:00:02.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'12144", 'reported_seq': 12228, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:58.018653+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T11:59:58.018653+0000', 'last_peered': '2024-09-18T11:59:58.018653+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T11:59:58.018653+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'2100", 'ondisk_log_start': "30'2100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30245481, 'num_objects': 7385, 'num_object_clones': 0, 'num_object_copies': 14770, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12143, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3850}, {'shards': '0,2', 'objects': 1156}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:02.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:00:02.186 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'12030", 'reported_seq': 12113, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:57.950594+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T11:59:57.950594+0000', 'last_peered': '2024-09-18T11:59:57.950594+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T11:59:57.950594+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'2000", 'ondisk_log_start': "30'2000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29361362, 'num_objects': 7170, 'num_object_clones': 0, 'num_object_copies': 14340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7170, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12028, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3577}, {'shards': '0,2', 'objects': 1164}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:02.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:00:02.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'12098", 'reported_seq': 12185, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T11:59:57.989218+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T11:59:57.989218+0000', 'last_peered': '2024-09-18T11:59:57.989218+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T11:59:57.989218+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'2000", 'ondisk_log_start': "30'2000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30370212, 'num_objects': 7418, 'num_object_clones': 0, 'num_object_copies': 14836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12095, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3895}, {'shards': '0,2', 'objects': 1186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:02.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:00:03.070 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:03.070 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33606/234924 objects degraded (14.305%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":403,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":117462,"data_bytes":481042527,"bytes_used":2030710784,"bytes_avail":287879581696,"bytes_total":289910292480,"degraded_objects":33606,"degraded_total":234924,"degraded_ratio":0.14305051846554631,"misplaced_objects":1175,"misplaced_total":234924,"misplaced_ratio":0.0050016175443973361,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":68250,"recovering_keys_per_sec":0,"num_objects_recovered":200,"num_bytes_recovered":819200,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":343},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:00:03.109 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:00:03.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:03.787 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:03.803 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-18T12:00:03.160027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":476164191,"num_objects":116271,"num_object_clones":0,"num_object_copies":232542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33606,"num_objects_misplaced":1175,"num_objects_unfound":0,"num_objects_dirty":116271,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178948,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7812,"num_bytes_recovered":31883900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160756,"ondisk_log_size":160756,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1980952,"kb_used_data":1058300,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281134568,"statfs":{"total":289910292480,"available":287881797632,"internally_reserved":0,"allocated":1083699200,"data_stored":1082929495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,0,7,11,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21741568,"num_objects":-5308,"num_object_clones":0,"num_object_copies":-10616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":-254,"num_objects_unfound":0,"num_objects_dirty":-5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5308,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002670"},"pg_stats":[{"pgid":"2.f","version":"41'12282","reported_seq":12355,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:02.844475+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:02.844475+0000","last_peered":"2024-09-18T12:00:02.844475+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:02.844475+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29712384,"num_objects":7254,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12282,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3655},{"shards":"0,2","objects":1085}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12118","reported_seq":12278,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.228663+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T11:59:58.228663+0000","last_peered":"2024-09-18T11:59:58.228663+0000","last_clean":"2024-09-18T11:59:58.228663+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T11:59:58.228663+0000","last_undegraded":"2024-09-18T11:59:58.228663+0000","last_fullsized":"2024-09-18T11:59:58.228663+0000","mapping_epoch":29,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29958144,"num_objects":7314,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12112","reported_seq":12300,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.201693+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T11:59:58.201693+0000","last_peered":"2024-09-18T11:59:58.201693+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T11:59:58.201693+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30494744,"num_objects":7446,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7446,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8970,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1204},{"shards":"2","objects":3910}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12015","reported_seq":12046,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.182285+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T11:59:58.182285+0000","last_peered":"2024-09-18T11:59:58.182285+0000","last_clean":"2024-09-18T11:59:58.182285+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T11:59:58.182285+0000","last_undegraded":"2024-09-18T11:59:58.182285+0000","last_fullsized":"2024-09-18T11:59:58.182285+0000","mapping_epoch":29,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29675520,"num_objects":7245,"num_object_clones":0,"num_object_copies":14490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12015,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12186","reported_seq":12379,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.065140+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:03.065140+0000","last_peered":"2024-09-18T12:00:03.065140+0000","last_clean":"2024-09-18T12:00:03.065140+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:03.065140+0000","last_undegraded":"2024-09-18T12:00:03.065140+0000","last_fullsized":"2024-09-18T12:00:03.065140+0000","mapping_epoch":29,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29741056,"num_objects":7261,"num_object_clones":0,"num_object_copies":14522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12209","reported_seq":12292,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:02.958032+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:02.958032+0000","last_peered":"2024-09-18T12:00:02.958032+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:02.958032+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29549778,"num_objects":7216,"num_object_clones":0,"num_object_copies":14432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12207,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3689},{"shards":"0,2","objects":1032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12170","reported_seq":12259,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.002049+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:03.002049+0000","last_peered":"2024-09-18T12:00:03.002049+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:03.002049+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29500626,"num_objects":7204,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3645},{"shards":"0,2","objects":1078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12414","reported_seq":12640,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.056661+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:03.056661+0000","last_peered":"2024-09-18T12:00:03.056661+0000","last_clean":"2024-09-18T12:00:03.056661+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:03.056661+0000","last_undegraded":"2024-09-18T12:00:03.056661+0000","last_fullsized":"2024-09-18T12:00:03.056661+0000","mapping_epoch":29,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29442048,"num_objects":7188,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12076","reported_seq":19206,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.570038+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T11:59:58.570038+0000","last_peered":"2024-09-18T11:59:58.570038+0000","last_clean":"2024-09-18T11:59:58.570038+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T11:59:58.570038+0000","last_undegraded":"2024-09-18T11:59:58.570038+0000","last_fullsized":"2024-09-18T11:59:58.570038+0000","mapping_epoch":38,"log_start":"30'2000","ondisk_log_start":"30'2000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30020818,"num_objects":7331,"num_object_clones":0,"num_object_copies":14662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12074,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12225","reported_seq":12305,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.043104+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:03.043104+0000","last_peered":"2024-09-18T12:00:03.043104+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:03.043104+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29578450,"num_objects":7223,"num_object_clones":0,"num_object_copies":14446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12223,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3691},{"shards":"0,2","objects":1034}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12149","reported_seq":12207,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.198148+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T11:59:58.198148+0000","last_peered":"2024-09-18T11:59:58.198148+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T11:59:58.198148+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29959378,"num_objects":7316,"num_object_clones":0,"num_object_copies":14632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12147,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1135},{"shards":"2","objects":3766}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12169","reported_seq":13945,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T11:59:58.246563+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T11:59:58.246563+0000","last_peered":"2024-09-18T11:59:58.246563+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T11:59:58.246563+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30019584,"num_objects":7329,"num_object_clones":0,"num_object_copies":14658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1637,"num_objects_misplaced":1175,"num_objects_unfound":0,"num_objects_dirty":7329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8649,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":783,"num_bytes_recovered":3198976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1175},{"shards":"2","objects":4517}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12277","reported_seq":12356,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:02.976788+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:02.976788+0000","last_peered":"2024-09-18T12:00:02.976788+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:02.976788+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29700713,"num_objects":7252,"num_object_clones":0,"num_object_copies":14504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12276,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3717},{"shards":"0,2","objects":1023}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12159","reported_seq":19307,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:58.548225+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T11:59:58.548225+0000","last_peered":"2024-09-18T11:59:58.548225+0000","last_clean":"2024-09-18T11:59:58.548225+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T11:59:58.548225+0000","last_undegraded":"2024-09-18T11:59:58.548225+0000","last_fullsized":"2024-09-18T11:59:58.548225+0000","mapping_epoch":40,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30160699,"num_objects":7366,"num_object_clones":0,"num_object_copies":14732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12156,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12155","reported_seq":12238,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:02.986302+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:02.986302+0000","last_peered":"2024-09-18T12:00:02.986302+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:02.986302+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28849362,"num_objects":7045,"num_object_clones":0,"num_object_copies":14090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12153,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3452},{"shards":"0,2","objects":1039}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12237","reported_seq":12324,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.048805+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:03.048805+0000","last_peered":"2024-09-18T12:00:03.048805+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:03.048805+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29800868,"num_objects":7279,"num_object_clones":0,"num_object_copies":14558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3756},{"shards":"0,2","objects":1047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":476164172,"num_objects":116269,"num_object_clones":0,"num_object_copies":232538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33606,"num_objects_misplaced":1175,"num_objects_unfound":0,"num_objects_dirty":116269,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178946,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7811,"num_bytes_recovered":31883881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1081786368,"data_stored":1081715602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051657,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650828,"kb_used_data":348804,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93721012,"statfs":{"total":96636764160,"available":95970316288,"internally_reserved":0,"allocated":357175296,"data_stored":356937180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.35399999999999998}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.40899999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855420,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616360,"kb_used_data":306012,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93755480,"statfs":{"total":96636764160,"available":96005611520,"internally_reserved":0,"allocated":313356288,"data_stored":313062365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.253},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":3.3359999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":4.9210000000000003},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.28399999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713764,"kb_used_data":403484,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93658076,"statfs":{"total":96636764160,"available":95905869824,"internally_reserved":0,"allocated":413167616,"data_stored":412929950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":2.1429999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":4.7850000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":3.9910000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":2.1720000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412127232,"data_stored":412123753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":312942592,"data_stored":312882856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356716544,"data_stored":356708993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:06.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:07.470 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:07.500 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:07.525 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-18T12:00:07.160976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":472461407,"num_objects":115367,"num_object_clones":0,"num_object_copies":230734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33655,"num_objects_misplaced":1033,"num_objects_unfound":0,"num_objects_dirty":115367,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179852,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7905,"num_bytes_recovered":32264828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1971584,"kb_used_data":1053084,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281143936,"statfs":{"total":289910292480,"available":287891390464,"internally_reserved":0,"allocated":1078358016,"data_stored":1077588311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,2,8,11,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17022976,"num_objects":-4156,"num_object_clones":0,"num_object_copies":-8312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":83,"num_objects_misplaced":-275,"num_objects_unfound":0,"num_objects_dirty":-4156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002744"},"pg_stats":[{"pgid":"2.f","version":"41'12282","reported_seq":12355,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:02.844475+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:02.844475+0000","last_peered":"2024-09-18T12:00:02.844475+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:02.844475+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29712384,"num_objects":7254,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12282,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3655},{"shards":"0,2","objects":1085}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12255","reported_seq":12415,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.237305+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:03.237305+0000","last_peered":"2024-09-18T12:00:03.237305+0000","last_clean":"2024-09-18T12:00:03.237305+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:03.237305+0000","last_undegraded":"2024-09-18T12:00:03.237305+0000","last_fullsized":"2024-09-18T12:00:03.237305+0000","mapping_epoch":29,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29396992,"num_objects":7177,"num_object_clones":0,"num_object_copies":14354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12234","reported_seq":12422,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.223615+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:03.223615+0000","last_peered":"2024-09-18T12:00:03.223615+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:03.223615+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29995032,"num_objects":7324,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9092,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1082},{"shards":"2","objects":3788}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12146","reported_seq":12177,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.219949+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:03.219949+0000","last_peered":"2024-09-18T12:00:03.219949+0000","last_clean":"2024-09-18T12:00:03.219949+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:03.219949+0000","last_undegraded":"2024-09-18T12:00:03.219949+0000","last_fullsized":"2024-09-18T12:00:03.219949+0000","mapping_epoch":29,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29138944,"num_objects":7114,"num_object_clones":0,"num_object_copies":14228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12146,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12186","reported_seq":12379,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.065140+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:03.065140+0000","last_peered":"2024-09-18T12:00:03.065140+0000","last_clean":"2024-09-18T12:00:03.065140+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:03.065140+0000","last_undegraded":"2024-09-18T12:00:03.065140+0000","last_fullsized":"2024-09-18T12:00:03.065140+0000","mapping_epoch":29,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29741056,"num_objects":7261,"num_object_clones":0,"num_object_copies":14522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12209","reported_seq":12292,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:02.958032+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:02.958032+0000","last_peered":"2024-09-18T12:00:02.958032+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:02.958032+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29549778,"num_objects":7216,"num_object_clones":0,"num_object_copies":14432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12207,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3689},{"shards":"0,2","objects":1032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12170","reported_seq":12259,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.002049+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:03.002049+0000","last_peered":"2024-09-18T12:00:03.002049+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:03.002049+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29500626,"num_objects":7204,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3645},{"shards":"0,2","objects":1078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12414","reported_seq":12640,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.056661+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:03.056661+0000","last_peered":"2024-09-18T12:00:03.056661+0000","last_clean":"2024-09-18T12:00:03.056661+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:03.056661+0000","last_undegraded":"2024-09-18T12:00:03.056661+0000","last_fullsized":"2024-09-18T12:00:03.056661+0000","mapping_epoch":29,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29442048,"num_objects":7188,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12210","reported_seq":19340,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.598404+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:03.598404+0000","last_peered":"2024-09-18T12:00:03.598404+0000","last_clean":"2024-09-18T12:00:03.598404+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:03.598404+0000","last_undegraded":"2024-09-18T12:00:03.598404+0000","last_fullsized":"2024-09-18T12:00:03.598404+0000","mapping_epoch":38,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29471954,"num_objects":7197,"num_object_clones":0,"num_object_copies":14394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12208,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12225","reported_seq":12305,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.043104+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:03.043104+0000","last_peered":"2024-09-18T12:00:03.043104+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:03.043104+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29578450,"num_objects":7223,"num_object_clones":0,"num_object_copies":14446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12223,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3691},{"shards":"0,2","objects":1034}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12279","reported_seq":12337,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.241300+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:03.241300+0000","last_peered":"2024-09-18T12:00:03.241300+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:03.241300+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29426898,"num_objects":7186,"num_object_clones":0,"num_object_copies":14372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12277,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1005},{"shards":"2","objects":3636}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12307","reported_seq":14271,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.241752+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:03.241752+0000","last_peered":"2024-09-18T12:00:03.241752+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:03.241752+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29454336,"num_objects":7191,"num_object_clones":0,"num_object_copies":14382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1686,"num_objects_misplaced":1033,"num_objects_unfound":0,"num_objects_dirty":7191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8787,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":876,"num_bytes_recovered":3579904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1033},{"shards":"2","objects":4472}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12277","reported_seq":12356,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:02.976788+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:02.976788+0000","last_peered":"2024-09-18T12:00:02.976788+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:02.976788+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29700713,"num_objects":7252,"num_object_clones":0,"num_object_copies":14504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12276,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3717},{"shards":"0,2","objects":1023}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12271","reported_seq":19419,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.559368+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:03.559368+0000","last_peered":"2024-09-18T12:00:03.559368+0000","last_clean":"2024-09-18T12:00:03.559368+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:03.559368+0000","last_undegraded":"2024-09-18T12:00:03.559368+0000","last_fullsized":"2024-09-18T12:00:03.559368+0000","mapping_epoch":40,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29701947,"num_objects":7254,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12268,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12155","reported_seq":12238,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:02.986302+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:02.986302+0000","last_peered":"2024-09-18T12:00:02.986302+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:02.986302+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28849362,"num_objects":7045,"num_object_clones":0,"num_object_copies":14090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12153,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3452},{"shards":"0,2","objects":1039}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12237","reported_seq":12324,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.048805+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:03.048805+0000","last_peered":"2024-09-18T12:00:03.048805+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:03.048805+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29800868,"num_objects":7279,"num_object_clones":0,"num_object_copies":14558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3756},{"shards":"0,2","objects":1047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":472461388,"num_objects":115365,"num_object_clones":0,"num_object_copies":230730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33655,"num_objects_misplaced":1033,"num_objects_unfound":0,"num_objects_dirty":115365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179850,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7904,"num_bytes_recovered":32264809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1076391936,"data_stored":1076321170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160757,"ondisk_log_size":160757,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051658,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650736,"kb_used_data":344540,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93721104,"statfs":{"total":96636764160,"available":95970410496,"internally_reserved":0,"allocated":352808960,"data_stored":352570844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.47099999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.378}]}]},{"osd":1,"up_from":35,"seq":150323855421,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607084,"kb_used_data":305060,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93764756,"statfs":{"total":96636764160,"available":96015110144,"internally_reserved":0,"allocated":312381440,"data_stored":312087517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":2.2959999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.38800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713764,"kb_used_data":403484,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93658076,"statfs":{"total":96636764160,"available":95905869824,"internally_reserved":0,"allocated":413167616,"data_stored":412929950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":2.1429999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":4.7850000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":3.9910000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":2.1720000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412127232,"data_stored":412123753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":311930880,"data_stored":311871144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":352333824,"data_stored":352326273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:07.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:08.236 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:08.246 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:08.259 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-18T12:00:07.160976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":472461407,"num_objects":115367,"num_object_clones":0,"num_object_copies":230734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33655,"num_objects_misplaced":1033,"num_objects_unfound":0,"num_objects_dirty":115367,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179852,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7905,"num_bytes_recovered":32264828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1971584,"kb_used_data":1053084,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281143936,"statfs":{"total":289910292480,"available":287891390464,"internally_reserved":0,"allocated":1078358016,"data_stored":1077588311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,2,8,11,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17022976,"num_objects":-4156,"num_object_clones":0,"num_object_copies":-8312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":83,"num_objects_misplaced":-275,"num_objects_unfound":0,"num_objects_dirty":-4156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002744"},"pg_stats":[{"pgid":"2.f","version":"41'12282","reported_seq":12355,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:02.844475+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:02.844475+0000","last_peered":"2024-09-18T12:00:02.844475+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:02.844475+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29712384,"num_objects":7254,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12282,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3655},{"shards":"0,2","objects":1085}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12255","reported_seq":12415,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.237305+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:03.237305+0000","last_peered":"2024-09-18T12:00:03.237305+0000","last_clean":"2024-09-18T12:00:03.237305+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:03.237305+0000","last_undegraded":"2024-09-18T12:00:03.237305+0000","last_fullsized":"2024-09-18T12:00:03.237305+0000","mapping_epoch":29,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29396992,"num_objects":7177,"num_object_clones":0,"num_object_copies":14354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12234","reported_seq":12422,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.223615+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:03.223615+0000","last_peered":"2024-09-18T12:00:03.223615+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:03.223615+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29995032,"num_objects":7324,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9092,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1082},{"shards":"2","objects":3788}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12146","reported_seq":12177,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.219949+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:03.219949+0000","last_peered":"2024-09-18T12:00:03.219949+0000","last_clean":"2024-09-18T12:00:03.219949+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:03.219949+0000","last_undegraded":"2024-09-18T12:00:03.219949+0000","last_fullsized":"2024-09-18T12:00:03.219949+0000","mapping_epoch":29,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29138944,"num_objects":7114,"num_object_clones":0,"num_object_copies":14228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12146,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12186","reported_seq":12379,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.065140+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:03.065140+0000","last_peered":"2024-09-18T12:00:03.065140+0000","last_clean":"2024-09-18T12:00:03.065140+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:03.065140+0000","last_undegraded":"2024-09-18T12:00:03.065140+0000","last_fullsized":"2024-09-18T12:00:03.065140+0000","mapping_epoch":29,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29741056,"num_objects":7261,"num_object_clones":0,"num_object_copies":14522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12209","reported_seq":12292,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:02.958032+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:02.958032+0000","last_peered":"2024-09-18T12:00:02.958032+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:02.958032+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29549778,"num_objects":7216,"num_object_clones":0,"num_object_copies":14432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12207,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3689},{"shards":"0,2","objects":1032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12170","reported_seq":12259,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.002049+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:03.002049+0000","last_peered":"2024-09-18T12:00:03.002049+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:03.002049+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29500626,"num_objects":7204,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3645},{"shards":"0,2","objects":1078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12414","reported_seq":12640,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.056661+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:03.056661+0000","last_peered":"2024-09-18T12:00:03.056661+0000","last_clean":"2024-09-18T12:00:03.056661+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:03.056661+0000","last_undegraded":"2024-09-18T12:00:03.056661+0000","last_fullsized":"2024-09-18T12:00:03.056661+0000","mapping_epoch":29,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29442048,"num_objects":7188,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12210","reported_seq":19340,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.598404+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:03.598404+0000","last_peered":"2024-09-18T12:00:03.598404+0000","last_clean":"2024-09-18T12:00:03.598404+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:03.598404+0000","last_undegraded":"2024-09-18T12:00:03.598404+0000","last_fullsized":"2024-09-18T12:00:03.598404+0000","mapping_epoch":38,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29471954,"num_objects":7197,"num_object_clones":0,"num_object_copies":14394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12208,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12225","reported_seq":12305,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.043104+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:03.043104+0000","last_peered":"2024-09-18T12:00:03.043104+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:03.043104+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29578450,"num_objects":7223,"num_object_clones":0,"num_object_copies":14446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12223,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3691},{"shards":"0,2","objects":1034}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12279","reported_seq":12337,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.241300+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:03.241300+0000","last_peered":"2024-09-18T12:00:03.241300+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:03.241300+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29426898,"num_objects":7186,"num_object_clones":0,"num_object_copies":14372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12277,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1005},{"shards":"2","objects":3636}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12307","reported_seq":14271,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.241752+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:03.241752+0000","last_peered":"2024-09-18T12:00:03.241752+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:03.241752+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29454336,"num_objects":7191,"num_object_clones":0,"num_object_copies":14382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1686,"num_objects_misplaced":1033,"num_objects_unfound":0,"num_objects_dirty":7191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8787,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":876,"num_bytes_recovered":3579904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1033},{"shards":"2","objects":4472}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12277","reported_seq":12356,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:02.976788+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:02.976788+0000","last_peered":"2024-09-18T12:00:02.976788+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:02.976788+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29700713,"num_objects":7252,"num_object_clones":0,"num_object_copies":14504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12276,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3717},{"shards":"0,2","objects":1023}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12271","reported_seq":19419,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:03.559368+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:03.559368+0000","last_peered":"2024-09-18T12:00:03.559368+0000","last_clean":"2024-09-18T12:00:03.559368+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:03.559368+0000","last_undegraded":"2024-09-18T12:00:03.559368+0000","last_fullsized":"2024-09-18T12:00:03.559368+0000","mapping_epoch":40,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29701947,"num_objects":7254,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12268,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12155","reported_seq":12238,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:02.986302+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:02.986302+0000","last_peered":"2024-09-18T12:00:02.986302+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:02.986302+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2100","ondisk_log_start":"30'2100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28849362,"num_objects":7045,"num_object_clones":0,"num_object_copies":14090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12153,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3452},{"shards":"0,2","objects":1039}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12237","reported_seq":12324,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:03.048805+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:03.048805+0000","last_peered":"2024-09-18T12:00:03.048805+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:03.048805+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29800868,"num_objects":7279,"num_object_clones":0,"num_object_copies":14558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3756},{"shards":"0,2","objects":1047}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":472461388,"num_objects":115365,"num_object_clones":0,"num_object_copies":230730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33655,"num_objects_misplaced":1033,"num_objects_unfound":0,"num_objects_dirty":115365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179850,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7904,"num_bytes_recovered":32264809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1076391936,"data_stored":1076321170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160757,"ondisk_log_size":160757,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051658,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650736,"kb_used_data":344540,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93721104,"statfs":{"total":96636764160,"available":95970410496,"internally_reserved":0,"allocated":352808960,"data_stored":352570844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.47099999999999997}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.378}]}]},{"osd":1,"up_from":35,"seq":150323855421,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607084,"kb_used_data":305060,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93764756,"statfs":{"total":96636764160,"available":96015110144,"internally_reserved":0,"allocated":312381440,"data_stored":312087517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":2.2959999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.38800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713764,"kb_used_data":403484,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93658076,"statfs":{"total":96636764160,"available":95905869824,"internally_reserved":0,"allocated":413167616,"data_stored":412929950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":2.1429999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":4.7850000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":3.9910000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":2.1720000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":412127232,"data_stored":412123753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":311930880,"data_stored":311871144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":352333824,"data_stored":352326273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:08.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:00:08.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'12282", 'reported_seq': 12355, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:02.844475+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:00:02.844475+0000', 'last_peered': '2024-09-18T12:00:02.844475+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:00:02.844475+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'2200", 'ondisk_log_start': "30'2200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29712384, 'num_objects': 7254, 'num_object_clones': 0, 'num_object_copies': 14508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12282, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3655}, {'shards': '0,2', 'objects': 1085}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:08.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:00:08.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'12234", 'reported_seq': 12422, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:03.223615+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:00:03.223615+0000', 'last_peered': '2024-09-18T12:00:03.223615+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:00:03.223615+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'2200", 'ondisk_log_start': "30'2200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29995032, 'num_objects': 7324, 'num_object_clones': 0, 'num_object_copies': 14648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7324, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9092, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1082}, {'shards': '2', 'objects': 3788}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:08.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:00:08.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'12209", 'reported_seq': 12292, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:02.958032+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:00:02.958032+0000', 'last_peered': '2024-09-18T12:00:02.958032+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:00:02.958032+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'2200", 'ondisk_log_start': "30'2200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29549778, 'num_objects': 7216, 'num_object_clones': 0, 'num_object_copies': 14432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12207, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3689}, {'shards': '0,2', 'objects': 1032}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:08.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:00:08.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'12170", 'reported_seq': 12259, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:03.002049+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:00:03.002049+0000', 'last_peered': '2024-09-18T12:00:03.002049+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:00:03.002049+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'2100", 'ondisk_log_start': "30'2100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29500626, 'num_objects': 7204, 'num_object_clones': 0, 'num_object_copies': 14408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12168, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3645}, {'shards': '0,2', 'objects': 1078}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:08.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:00:08.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'12225", 'reported_seq': 12305, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:03.043104+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:00:03.043104+0000', 'last_peered': '2024-09-18T12:00:03.043104+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:00:03.043104+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'2200", 'ondisk_log_start': "30'2200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29578450, 'num_objects': 7223, 'num_object_clones': 0, 'num_object_copies': 14446, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7223, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12223, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3691}, {'shards': '0,2', 'objects': 1034}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:08.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:00:08.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'12279", 'reported_seq': 12337, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:03.241300+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:00:03.241300+0000', 'last_peered': '2024-09-18T12:00:03.241300+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:00:03.241300+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'2200", 'ondisk_log_start': "30'2200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29426898, 'num_objects': 7186, 'num_object_clones': 0, 'num_object_copies': 14372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12277, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1005}, {'shards': '2', 'objects': 3636}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:08.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:00:08.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'12307", 'reported_seq': 14271, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:03.241752+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:00:03.241752+0000', 'last_peered': '2024-09-18T12:00:03.241752+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:00:03.241752+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'2300", 'ondisk_log_start': "30'2300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29454336, 'num_objects': 7191, 'num_object_clones': 0, 'num_object_copies': 14382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1686, 'num_objects_misplaced': 1033, 'num_objects_unfound': 0, 'num_objects_dirty': 7191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8787, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 876, 'num_bytes_recovered': 3579904, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1033}, {'shards': '2', 'objects': 4472}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:08.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:00:08.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'12277", 'reported_seq': 12356, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:02.976788+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:00:02.976788+0000', 'last_peered': '2024-09-18T12:00:02.976788+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:00:02.976788+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'2200", 'ondisk_log_start': "30'2200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29700713, 'num_objects': 7252, 'num_object_clones': 0, 'num_object_copies': 14504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12276, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3717}, {'shards': '0,2', 'objects': 1023}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:08.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:00:08.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'12155", 'reported_seq': 12238, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:02.986302+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:00:02.986302+0000', 'last_peered': '2024-09-18T12:00:02.986302+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:00:02.986302+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'2100", 'ondisk_log_start': "30'2100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28849362, 'num_objects': 7045, 'num_object_clones': 0, 'num_object_copies': 14090, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7045, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12153, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3452}, {'shards': '0,2', 'objects': 1039}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:08.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:00:08.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'12237", 'reported_seq': 12324, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:03.048805+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:00:03.048805+0000', 'last_peered': '2024-09-18T12:00:03.048805+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:00:03.048805+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'2200", 'ondisk_log_start': "30'2200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29800868, 'num_objects': 7279, 'num_object_clones': 0, 'num_object_copies': 14558, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12234, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3756}, {'shards': '0,2', 'objects': 1047}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:08.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:00:09.318 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:09.318 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33655/230734 objects degraded (14.586%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":409,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":115367,"data_bytes":472461407,"bytes_used":2018902016,"bytes_avail":287891390464,"bytes_total":289910292480,"degraded_objects":33655,"degraded_total":230734,"degraded_ratio":0.14586060138514481,"misplaced_objects":1033,"misplaced_total":230734,"misplaced_ratio":0.0044770168245685509,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":65521,"recovering_keys_per_sec":0,"num_objects_recovered":192,"num_bytes_recovered":786432,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":346},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:00:09.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:00:09.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:09.985 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:10.022 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:10.044 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-18T12:00:09.165826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":463806559,"num_objects":113254,"num_object_clones":0,"num_object_copies":226508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33685,"num_objects_misplaced":899,"num_objects_unfound":0,"num_objects_dirty":113254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181965,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8009,"num_bytes_recovered":32690812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160773,"ondisk_log_size":160773,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1955904,"kb_used_data":1041572,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281159616,"statfs":{"total":289910292480,"available":287907446784,"internally_reserved":0,"allocated":1066569728,"data_stored":1065800023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,7,5,11,13,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25677824,"num_objects":-6269,"num_object_clones":0,"num_object_copies":-12538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":113,"num_objects_misplaced":-409,"num_objects_unfound":0,"num_objects_dirty":-6269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6269,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007132"},"pg_stats":[{"pgid":"2.f","version":"41'12439","reported_seq":12512,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.023262+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:08.023262+0000","last_peered":"2024-09-18T12:00:08.023262+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:08.023262+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29069312,"num_objects":7097,"num_object_clones":0,"num_object_copies":14194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12439,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3498},{"shards":"0,2","objects":928}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12370","reported_seq":12530,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.128453+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:08.128453+0000","last_peered":"2024-09-18T12:00:08.128453+0000","last_clean":"2024-09-18T12:00:08.128453+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:08.128453+0000","last_undegraded":"2024-09-18T12:00:08.128453+0000","last_fullsized":"2024-09-18T12:00:08.128453+0000","mapping_epoch":29,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28925952,"num_objects":7062,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9562,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12374","reported_seq":12559,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.259201+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:08.259201+0000","last_peered":"2024-09-18T12:00:08.259201+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:08.259201+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29421592,"num_objects":7184,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7184,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9232,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":942},{"shards":"2","objects":3648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12283","reported_seq":12314,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.205402+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:08.205402+0000","last_peered":"2024-09-18T12:00:08.205402+0000","last_clean":"2024-09-18T12:00:08.205402+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:08.205402+0000","last_undegraded":"2024-09-18T12:00:08.205402+0000","last_fullsized":"2024-09-18T12:00:08.205402+0000","mapping_epoch":29,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28577792,"num_objects":6977,"num_object_clones":0,"num_object_copies":13954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12283,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12319","reported_seq":12510,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.044905+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:08.044905+0000","last_peered":"2024-09-18T12:00:08.044905+0000","last_clean":"2024-09-18T12:00:08.044905+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:08.044905+0000","last_undegraded":"2024-09-18T12:00:08.044905+0000","last_fullsized":"2024-09-18T12:00:08.044905+0000","mapping_epoch":29,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29196288,"num_objects":7128,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9484,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12338","reported_seq":12421,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.068511+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:08.068511+0000","last_peered":"2024-09-18T12:00:08.068511+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:08.068511+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29021394,"num_objects":7087,"num_object_clones":0,"num_object_copies":14174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12336,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3560},{"shards":"0,2","objects":903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12306","reported_seq":12395,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.024403+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:08.024403+0000","last_peered":"2024-09-18T12:00:08.024403+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:08.024403+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28943570,"num_objects":7068,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12304,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3509},{"shards":"0,2","objects":942}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12537","reported_seq":12764,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:07.749187+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:07.749187+0000","last_peered":"2024-09-18T12:00:07.749187+0000","last_clean":"2024-09-18T12:00:07.749187+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:07.749187+0000","last_undegraded":"2024-09-18T12:00:07.749187+0000","last_fullsized":"2024-09-18T12:00:07.749187+0000","mapping_epoch":29,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28938240,"num_objects":7065,"num_object_clones":0,"num_object_copies":14130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12325","reported_seq":19455,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.653559+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:08.653559+0000","last_peered":"2024-09-18T12:00:08.653559+0000","last_clean":"2024-09-18T12:00:08.653559+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:08.653559+0000","last_undegraded":"2024-09-18T12:00:08.653559+0000","last_fullsized":"2024-09-18T12:00:08.653559+0000","mapping_epoch":38,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29000914,"num_objects":7082,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12360","reported_seq":12438,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.040774+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:08.040774+0000","last_peered":"2024-09-18T12:00:08.040774+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:08.040774+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29025490,"num_objects":7088,"num_object_clones":0,"num_object_copies":14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12358,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3556},{"shards":"0,2","objects":899}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12400","reported_seq":12455,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.150080+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:08.150080+0000","last_peered":"2024-09-18T12:00:08.150080+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:08.150080+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28931282,"num_objects":7065,"num_object_clones":0,"num_object_copies":14130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12398,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":884},{"shards":"2","objects":3515}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12439","reported_seq":14611,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.268188+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:08.268188+0000","last_peered":"2024-09-18T12:00:08.268188+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:08.268188+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28913664,"num_objects":7059,"num_object_clones":0,"num_object_copies":14118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":899,"num_objects_unfound":0,"num_objects_dirty":7059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8919,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":4005888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":899},{"shards":"2","objects":4444}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12404","reported_seq":12487,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.039938+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:08.039938+0000","last_peered":"2024-09-18T12:00:08.039938+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:08.039938+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29180521,"num_objects":7125,"num_object_clones":0,"num_object_copies":14250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12403,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3590},{"shards":"0,2","objects":896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12427","reported_seq":19575,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.501287+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:08.501287+0000","last_peered":"2024-09-18T12:00:08.501287+0000","last_clean":"2024-09-18T12:00:08.501287+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:08.501287+0000","last_undegraded":"2024-09-18T12:00:08.501287+0000","last_fullsized":"2024-09-18T12:00:08.501287+0000","mapping_epoch":40,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062971,"num_objects":7098,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12424,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12290","reported_seq":12373,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.044684+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:08.044684+0000","last_peered":"2024-09-18T12:00:08.044684+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:08.044684+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28296402,"num_objects":6910,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12288,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3317},{"shards":"0,2","objects":904}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12359","reported_seq":12446,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.021491+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:08.021491+0000","last_peered":"2024-09-18T12:00:08.021491+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:08.021491+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29301156,"num_objects":7157,"num_object_clones":0,"num_object_copies":14314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12356,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3634},{"shards":"0,2","objects":925}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":463806540,"num_objects":113252,"num_object_clones":0,"num_object_copies":226504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33685,"num_objects_misplaced":899,"num_objects_unfound":0,"num_objects_dirty":113252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181963,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8008,"num_bytes_recovered":32690793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064574976,"data_stored":1064504210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160770,"ondisk_log_size":160770,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051659,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650804,"kb_used_data":340456,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93721036,"statfs":{"total":96636764160,"available":95970340864,"internally_reserved":0,"allocated":348626944,"data_stored":348388828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.38800000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.76400000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855422,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610144,"kb_used_data":303960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93761696,"statfs":{"total":96636764160,"available":96011976704,"internally_reserved":0,"allocated":311255040,"data_stored":310961117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,7,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.219}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.35599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694956,"kb_used_data":397156,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93676884,"statfs":{"total":96636764160,"available":95925129216,"internally_reserved":0,"allocated":406687744,"data_stored":406450078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,4,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.35399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":405561344,"data_stored":405557865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":310841344,"data_stored":310781608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":348172288,"data_stored":348164737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:13.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:13.848 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:13.874 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:13.881 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-09-18T12:00:13.167497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":458948703,"num_objects":112068,"num_object_clones":0,"num_object_copies":224136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33685,"num_objects_misplaced":899,"num_objects_unfound":0,"num_objects_dirty":112068,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183151,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8009,"num_bytes_recovered":32690812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160959,"ondisk_log_size":160959,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1958084,"kb_used_data":1035432,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281157436,"statfs":{"total":289910292480,"available":287905214464,"internally_reserved":0,"allocated":1060282368,"data_stored":1059512663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,0,6,3,9,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22093824,"num_objects":-5394,"num_object_clones":0,"num_object_copies":-10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":79,"num_objects_misplaced":-276,"num_objects_unfound":0,"num_objects_dirty":-5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5394,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":806912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007850"},"pg_stats":[{"pgid":"2.f","version":"41'12571","reported_seq":12644,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.035285+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:13.035285+0000","last_peered":"2024-09-18T12:00:13.035285+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:13.035285+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28528640,"num_objects":6965,"num_object_clones":0,"num_object_copies":13930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3366},{"shards":"0,2","objects":796}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12370","reported_seq":12530,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.128453+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:08.128453+0000","last_peered":"2024-09-18T12:00:08.128453+0000","last_clean":"2024-09-18T12:00:08.128453+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:08.128453+0000","last_undegraded":"2024-09-18T12:00:08.128453+0000","last_fullsized":"2024-09-18T12:00:08.128453+0000","mapping_epoch":29,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28925952,"num_objects":7062,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9562,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12374","reported_seq":12559,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.259201+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:08.259201+0000","last_peered":"2024-09-18T12:00:08.259201+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:08.259201+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29421592,"num_objects":7184,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7184,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9232,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":942},{"shards":"2","objects":3648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12283","reported_seq":12314,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.205402+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:08.205402+0000","last_peered":"2024-09-18T12:00:08.205402+0000","last_clean":"2024-09-18T12:00:08.205402+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:08.205402+0000","last_undegraded":"2024-09-18T12:00:08.205402+0000","last_fullsized":"2024-09-18T12:00:08.205402+0000","mapping_epoch":29,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28577792,"num_objects":6977,"num_object_clones":0,"num_object_copies":13954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12283,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12442","reported_seq":12635,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:12.983144+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:12.983144+0000","last_peered":"2024-09-18T12:00:12.983144+0000","last_clean":"2024-09-18T12:00:12.983144+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:12.983144+0000","last_undegraded":"2024-09-18T12:00:12.983144+0000","last_fullsized":"2024-09-18T12:00:12.983144+0000","mapping_epoch":29,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28692480,"num_objects":7005,"num_object_clones":0,"num_object_copies":14010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9607,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12479","reported_seq":12561,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.095225+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:13.095225+0000","last_peered":"2024-09-18T12:00:13.095225+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:13.095225+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28443858,"num_objects":6946,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3419},{"shards":"0,2","objects":762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12440","reported_seq":12529,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:12.958403+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:12.958403+0000","last_peered":"2024-09-18T12:00:12.958403+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:12.958403+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28394706,"num_objects":6934,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12438,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3375},{"shards":"0,2","objects":808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12680","reported_seq":12907,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:12.859912+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:12.859912+0000","last_peered":"2024-09-18T12:00:12.859912+0000","last_clean":"2024-09-18T12:00:12.859912+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:12.859912+0000","last_undegraded":"2024-09-18T12:00:12.859912+0000","last_fullsized":"2024-09-18T12:00:12.859912+0000","mapping_epoch":29,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28352512,"num_objects":6922,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8997,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12325","reported_seq":19455,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.653559+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:08.653559+0000","last_peered":"2024-09-18T12:00:08.653559+0000","last_clean":"2024-09-18T12:00:08.653559+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:08.653559+0000","last_undegraded":"2024-09-18T12:00:08.653559+0000","last_fullsized":"2024-09-18T12:00:08.653559+0000","mapping_epoch":38,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29000914,"num_objects":7082,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12488","reported_seq":12567,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.081327+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:13.081327+0000","last_peered":"2024-09-18T12:00:13.081327+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:13.081327+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28501202,"num_objects":6960,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12486,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3428},{"shards":"0,2","objects":771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12400","reported_seq":12455,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.150080+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:08.150080+0000","last_peered":"2024-09-18T12:00:08.150080+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:08.150080+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28931282,"num_objects":7065,"num_object_clones":0,"num_object_copies":14130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12398,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":884},{"shards":"2","objects":3515}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12439","reported_seq":14611,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.268188+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:08.268188+0000","last_peered":"2024-09-18T12:00:08.268188+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:08.268188+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28913664,"num_objects":7059,"num_object_clones":0,"num_object_copies":14118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":899,"num_objects_unfound":0,"num_objects_dirty":7059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8919,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":4005888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":899},{"shards":"2","objects":4444}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12530","reported_seq":12614,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:12.922800+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:12.922800+0000","last_peered":"2024-09-18T12:00:12.922800+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:12.922800+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28664425,"num_objects":6999,"num_object_clones":0,"num_object_copies":13998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12529,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3464},{"shards":"0,2","objects":770}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12427","reported_seq":19575,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.501287+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:08.501287+0000","last_peered":"2024-09-18T12:00:08.501287+0000","last_clean":"2024-09-18T12:00:08.501287+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:08.501287+0000","last_undegraded":"2024-09-18T12:00:08.501287+0000","last_fullsized":"2024-09-18T12:00:08.501287+0000","mapping_epoch":40,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062971,"num_objects":7098,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12424,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12418","reported_seq":12501,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.063644+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:13.063644+0000","last_peered":"2024-09-18T12:00:13.063644+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:13.063644+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27772114,"num_objects":6782,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3189},{"shards":"0,2","objects":776}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12490","reported_seq":12577,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.091106+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:13.091106+0000","last_peered":"2024-09-18T12:00:13.091106+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:13.091106+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28764580,"num_objects":7026,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12487,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3503},{"shards":"0,2","objects":794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":458948684,"num_objects":112066,"num_object_clones":0,"num_object_copies":224132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33685,"num_objects_misplaced":899,"num_objects_unfound":0,"num_objects_dirty":112066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183149,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8008,"num_bytes_recovered":32690793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1058205696,"data_stored":1058134930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051659,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650804,"kb_used_data":340456,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93721036,"statfs":{"total":96636764160,"available":95970340864,"internally_reserved":0,"allocated":348626944,"data_stored":348388828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.38800000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.76400000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855422,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610144,"kb_used_data":303960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93761696,"statfs":{"total":96636764160,"available":96011976704,"internally_reserved":0,"allocated":311255040,"data_stored":310961117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,7,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.219}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.35599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697136,"kb_used_data":391016,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93674704,"statfs":{"total":96636764160,"available":95922896896,"internally_reserved":0,"allocated":400400384,"data_stored":400162718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.314},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":8.3460000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.28299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":399192064,"data_stored":399188585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":310841344,"data_stored":310781608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":348172288,"data_stored":348164737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:13.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:14.531 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:14.554 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:14.563 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-09-18T12:00:13.167497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":458948703,"num_objects":112068,"num_object_clones":0,"num_object_copies":224136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33685,"num_objects_misplaced":899,"num_objects_unfound":0,"num_objects_dirty":112068,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183151,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8009,"num_bytes_recovered":32690812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160959,"ondisk_log_size":160959,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1958084,"kb_used_data":1035432,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281157436,"statfs":{"total":289910292480,"available":287905214464,"internally_reserved":0,"allocated":1060282368,"data_stored":1059512663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,0,6,3,9,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22093824,"num_objects":-5394,"num_object_clones":0,"num_object_copies":-10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":79,"num_objects_misplaced":-276,"num_objects_unfound":0,"num_objects_dirty":-5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5394,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":806912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007850"},"pg_stats":[{"pgid":"2.f","version":"41'12571","reported_seq":12644,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.035285+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:13.035285+0000","last_peered":"2024-09-18T12:00:13.035285+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:13.035285+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28528640,"num_objects":6965,"num_object_clones":0,"num_object_copies":13930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3366},{"shards":"0,2","objects":796}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12370","reported_seq":12530,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.128453+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:08.128453+0000","last_peered":"2024-09-18T12:00:08.128453+0000","last_clean":"2024-09-18T12:00:08.128453+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:08.128453+0000","last_undegraded":"2024-09-18T12:00:08.128453+0000","last_fullsized":"2024-09-18T12:00:08.128453+0000","mapping_epoch":29,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28925952,"num_objects":7062,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9562,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12374","reported_seq":12559,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.259201+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:08.259201+0000","last_peered":"2024-09-18T12:00:08.259201+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:08.259201+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29421592,"num_objects":7184,"num_object_clones":0,"num_object_copies":14368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7184,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9232,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":942},{"shards":"2","objects":3648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12283","reported_seq":12314,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.205402+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:08.205402+0000","last_peered":"2024-09-18T12:00:08.205402+0000","last_clean":"2024-09-18T12:00:08.205402+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:08.205402+0000","last_undegraded":"2024-09-18T12:00:08.205402+0000","last_fullsized":"2024-09-18T12:00:08.205402+0000","mapping_epoch":29,"log_start":"30'2200","ondisk_log_start":"30'2200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28577792,"num_objects":6977,"num_object_clones":0,"num_object_copies":13954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12283,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12442","reported_seq":12635,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:12.983144+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:12.983144+0000","last_peered":"2024-09-18T12:00:12.983144+0000","last_clean":"2024-09-18T12:00:12.983144+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:12.983144+0000","last_undegraded":"2024-09-18T12:00:12.983144+0000","last_fullsized":"2024-09-18T12:00:12.983144+0000","mapping_epoch":29,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28692480,"num_objects":7005,"num_object_clones":0,"num_object_copies":14010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9607,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12479","reported_seq":12561,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.095225+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:13.095225+0000","last_peered":"2024-09-18T12:00:13.095225+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:13.095225+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28443858,"num_objects":6946,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3419},{"shards":"0,2","objects":762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12440","reported_seq":12529,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:12.958403+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:12.958403+0000","last_peered":"2024-09-18T12:00:12.958403+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:12.958403+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28394706,"num_objects":6934,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12438,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3375},{"shards":"0,2","objects":808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12680","reported_seq":12907,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:12.859912+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:12.859912+0000","last_peered":"2024-09-18T12:00:12.859912+0000","last_clean":"2024-09-18T12:00:12.859912+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:12.859912+0000","last_undegraded":"2024-09-18T12:00:12.859912+0000","last_fullsized":"2024-09-18T12:00:12.859912+0000","mapping_epoch":29,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28352512,"num_objects":6922,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8997,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12325","reported_seq":19455,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.653559+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:08.653559+0000","last_peered":"2024-09-18T12:00:08.653559+0000","last_clean":"2024-09-18T12:00:08.653559+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:08.653559+0000","last_undegraded":"2024-09-18T12:00:08.653559+0000","last_fullsized":"2024-09-18T12:00:08.653559+0000","mapping_epoch":38,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29000914,"num_objects":7082,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12488","reported_seq":12567,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.081327+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:13.081327+0000","last_peered":"2024-09-18T12:00:13.081327+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:13.081327+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28501202,"num_objects":6960,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12486,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3428},{"shards":"0,2","objects":771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12400","reported_seq":12455,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.150080+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:08.150080+0000","last_peered":"2024-09-18T12:00:08.150080+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:08.150080+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2300","ondisk_log_start":"30'2300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28931282,"num_objects":7065,"num_object_clones":0,"num_object_copies":14130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12398,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":884},{"shards":"2","objects":3515}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12439","reported_seq":14611,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:08.268188+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:08.268188+0000","last_peered":"2024-09-18T12:00:08.268188+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:08.268188+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28913664,"num_objects":7059,"num_object_clones":0,"num_object_copies":14118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":899,"num_objects_unfound":0,"num_objects_dirty":7059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8919,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":4005888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":899},{"shards":"2","objects":4444}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12530","reported_seq":12614,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:12.922800+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:12.922800+0000","last_peered":"2024-09-18T12:00:12.922800+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:12.922800+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28664425,"num_objects":6999,"num_object_clones":0,"num_object_copies":13998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12529,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3464},{"shards":"0,2","objects":770}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12427","reported_seq":19575,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:08.501287+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:08.501287+0000","last_peered":"2024-09-18T12:00:08.501287+0000","last_clean":"2024-09-18T12:00:08.501287+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:08.501287+0000","last_undegraded":"2024-09-18T12:00:08.501287+0000","last_fullsized":"2024-09-18T12:00:08.501287+0000","mapping_epoch":40,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29062971,"num_objects":7098,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12424,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12418","reported_seq":12501,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.063644+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:13.063644+0000","last_peered":"2024-09-18T12:00:13.063644+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:13.063644+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27772114,"num_objects":6782,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3189},{"shards":"0,2","objects":776}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12490","reported_seq":12577,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.091106+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:13.091106+0000","last_peered":"2024-09-18T12:00:13.091106+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:13.091106+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28764580,"num_objects":7026,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12487,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3503},{"shards":"0,2","objects":794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":458948684,"num_objects":112066,"num_object_clones":0,"num_object_copies":224132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33685,"num_objects_misplaced":899,"num_objects_unfound":0,"num_objects_dirty":112066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183149,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8008,"num_bytes_recovered":32690793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1058205696,"data_stored":1058134930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160956,"ondisk_log_size":160956,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051659,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650804,"kb_used_data":340456,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93721036,"statfs":{"total":96636764160,"available":95970340864,"internally_reserved":0,"allocated":348626944,"data_stored":348388828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.38800000000000001}]},{"osd":1,"last update":"Wed Sep 18 11:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.5580000000000001,"15min":1.9079999999999999},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.9569999999999999,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.0580000000000001,"15min":0.93200000000000005},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":0.76400000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855422,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610144,"kb_used_data":303960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93761696,"statfs":{"total":96636764160,"available":96011976704,"internally_reserved":0,"allocated":311255040,"data_stored":310961117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,7,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.443,"15min":0.46300000000000002},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.3730000000000002,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.405,"15min":1.121},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":12.244999999999999,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.219}]},{"osd":2,"last update":"Wed Sep 18 11:59:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.3280000000000001,"15min":1.0409999999999999},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.147,"5min":11.147,"15min":11.147},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.87,"15min":0.70799999999999996},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4939999999999998,"5min":11.996,"15min":11.996},"last":0.35599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697136,"kb_used_data":391016,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93674704,"statfs":{"total":96636764160,"available":95922896896,"internally_reserved":0,"allocated":400400384,"data_stored":400162718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.314},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":8.3460000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.28299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":399192064,"data_stored":399188585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":310841344,"data_stored":310781608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":348172288,"data_stored":348164737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:14.564 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:00:14.564 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'12571", 'reported_seq': 12644, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:13.035285+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:00:13.035285+0000', 'last_peered': '2024-09-18T12:00:13.035285+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:00:13.035285+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'2500", 'ondisk_log_start': "30'2500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28528640, 'num_objects': 6965, 'num_object_clones': 0, 'num_object_copies': 13930, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12571, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3366}, {'shards': '0,2', 'objects': 796}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:14.564 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:00:14.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'12374", 'reported_seq': 12559, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:08.259201+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:00:08.259201+0000', 'last_peered': '2024-09-18T12:00:08.259201+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:00:08.259201+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'2300", 'ondisk_log_start': "30'2300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29421592, 'num_objects': 7184, 'num_object_clones': 0, 'num_object_copies': 14368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7184, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9232, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 942}, {'shards': '2', 'objects': 3648}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:14.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:00:14.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'12479", 'reported_seq': 12561, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:13.095225+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:00:13.095225+0000', 'last_peered': '2024-09-18T12:00:13.095225+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:00:13.095225+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'2400", 'ondisk_log_start': "30'2400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28443858, 'num_objects': 6946, 'num_object_clones': 0, 'num_object_copies': 13892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12477, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3419}, {'shards': '0,2', 'objects': 762}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:14.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:00:14.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'12440", 'reported_seq': 12529, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:12.958403+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:00:12.958403+0000', 'last_peered': '2024-09-18T12:00:12.958403+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:00:12.958403+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'2400", 'ondisk_log_start': "30'2400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28394706, 'num_objects': 6934, 'num_object_clones': 0, 'num_object_copies': 13868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12438, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3375}, {'shards': '0,2', 'objects': 808}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:14.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:00:14.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'12488", 'reported_seq': 12567, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:13.081327+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:00:13.081327+0000', 'last_peered': '2024-09-18T12:00:13.081327+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:00:13.081327+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'2400", 'ondisk_log_start': "30'2400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28501202, 'num_objects': 6960, 'num_object_clones': 0, 'num_object_copies': 13920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12486, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3428}, {'shards': '0,2', 'objects': 771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:14.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:00:14.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'12400", 'reported_seq': 12455, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:08.150080+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:00:08.150080+0000', 'last_peered': '2024-09-18T12:00:08.150080+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:00:08.150080+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'2300", 'ondisk_log_start': "30'2300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28931282, 'num_objects': 7065, 'num_object_clones': 0, 'num_object_copies': 14130, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12398, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 884}, {'shards': '2', 'objects': 3515}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:14.565 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:00:14.566 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'12439", 'reported_seq': 14611, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:08.268188+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:00:08.268188+0000', 'last_peered': '2024-09-18T12:00:08.268188+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:00:08.268188+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'2400", 'ondisk_log_start': "30'2400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28913664, 'num_objects': 7059, 'num_object_clones': 0, 'num_object_copies': 14118, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1716, 'num_objects_misplaced': 899, 'num_objects_unfound': 0, 'num_objects_dirty': 7059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8919, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 980, 'num_bytes_recovered': 4005888, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 899}, {'shards': '2', 'objects': 4444}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:14.566 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:00:14.566 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'12530", 'reported_seq': 12614, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:12.922800+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:00:12.922800+0000', 'last_peered': '2024-09-18T12:00:12.922800+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:00:12.922800+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'2500", 'ondisk_log_start': "30'2500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28664425, 'num_objects': 6999, 'num_object_clones': 0, 'num_object_copies': 13998, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12529, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3464}, {'shards': '0,2', 'objects': 770}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:14.566 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:00:14.566 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'12418", 'reported_seq': 12501, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:13.063644+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:00:13.063644+0000', 'last_peered': '2024-09-18T12:00:13.063644+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:00:13.063644+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'2400", 'ondisk_log_start': "30'2400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27772114, 'num_objects': 6782, 'num_object_clones': 0, 'num_object_copies': 13564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12416, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3189}, {'shards': '0,2', 'objects': 776}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:14.566 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:00:14.566 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'12490", 'reported_seq': 12577, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:13.091106+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:00:13.091106+0000', 'last_peered': '2024-09-18T12:00:13.091106+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:00:13.091106+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'2400", 'ondisk_log_start': "30'2400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28764580, 'num_objects': 7026, 'num_object_clones': 0, 'num_object_copies': 14052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12487, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3503}, {'shards': '0,2', 'objects': 794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:14.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:00:15.446 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:15.446 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33685/224136 objects degraded (15.029%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":415,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":112068,"data_bytes":458948703,"bytes_used":2005078016,"bytes_avail":287905214464,"bytes_total":289910292480,"degraded_objects":33685,"degraded_total":224136,"degraded_ratio":0.15028821786772317,"misplaced_objects":899,"misplaced_total":224136,"misplaced_ratio":0.0040109576328657597,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":67198,"recovering_keys_per_sec":0,"num_objects_recovered":197,"num_bytes_recovered":806912,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":449},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:00:15.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:00:15.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:16.196 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:16.202 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:16.215 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-18T12:00:15.167977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":455069791,"num_objects":111121,"num_object_clones":0,"num_object_copies":222242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33747,"num_objects_misplaced":746,"num_objects_unfound":0,"num_objects_dirty":111121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184098,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8100,"num_bytes_recovered":33063548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1948660,"kb_used_data":1030168,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281166860,"statfs":{"total":289910292480,"available":287914864640,"internally_reserved":0,"allocated":1054892032,"data_stored":1054122327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,6,2,7,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21094400,"num_objects":-5150,"num_object_clones":0,"num_object_copies":-10300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":141,"num_objects_misplaced":-429,"num_objects_unfound":0,"num_objects_dirty":-5150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5150,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007949"},"pg_stats":[{"pgid":"2.f","version":"41'12571","reported_seq":12644,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.035285+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:13.035285+0000","last_peered":"2024-09-18T12:00:13.035285+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:13.035285+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28528640,"num_objects":6965,"num_object_clones":0,"num_object_copies":13930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3366},{"shards":"0,2","objects":796}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12479","reported_seq":12639,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:13.318006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:13.318006+0000","last_peered":"2024-09-18T12:00:13.318006+0000","last_clean":"2024-09-18T12:00:13.318006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:13.318006+0000","last_undegraded":"2024-09-18T12:00:13.318006+0000","last_fullsized":"2024-09-18T12:00:13.318006+0000","mapping_epoch":29,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28479488,"num_objects":6953,"num_object_clones":0,"num_object_copies":13906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9671,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12526","reported_seq":12714,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.316152+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:13.316152+0000","last_peered":"2024-09-18T12:00:13.316152+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:13.316152+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28799000,"num_objects":7032,"num_object_clones":0,"num_object_copies":14064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7032,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9384,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":790},{"shards":"2","objects":3496}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12416","reported_seq":12447,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:13.176326+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:13.176326+0000","last_peered":"2024-09-18T12:00:13.176326+0000","last_clean":"2024-09-18T12:00:13.176326+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:13.176326+0000","last_undegraded":"2024-09-18T12:00:13.176326+0000","last_fullsized":"2024-09-18T12:00:13.176326+0000","mapping_epoch":29,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28033024,"num_objects":6844,"num_object_clones":0,"num_object_copies":13688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12442","reported_seq":12635,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:12.983144+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:12.983144+0000","last_peered":"2024-09-18T12:00:12.983144+0000","last_clean":"2024-09-18T12:00:12.983144+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:12.983144+0000","last_undegraded":"2024-09-18T12:00:12.983144+0000","last_fullsized":"2024-09-18T12:00:12.983144+0000","mapping_epoch":29,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28692480,"num_objects":7005,"num_object_clones":0,"num_object_copies":14010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9607,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12479","reported_seq":12561,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.095225+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:13.095225+0000","last_peered":"2024-09-18T12:00:13.095225+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:13.095225+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28443858,"num_objects":6946,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3419},{"shards":"0,2","objects":762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12440","reported_seq":12529,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:12.958403+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:12.958403+0000","last_peered":"2024-09-18T12:00:12.958403+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:12.958403+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28394706,"num_objects":6934,"num_object_clones":0,"num_object_copies":13868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12438,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3375},{"shards":"0,2","objects":808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12680","reported_seq":12907,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:12.859912+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:12.859912+0000","last_peered":"2024-09-18T12:00:12.859912+0000","last_clean":"2024-09-18T12:00:12.859912+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:12.859912+0000","last_undegraded":"2024-09-18T12:00:12.859912+0000","last_fullsized":"2024-09-18T12:00:12.859912+0000","mapping_epoch":29,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28352512,"num_objects":6922,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8997,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12451","reported_seq":19579,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:13.685939+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:13.685939+0000","last_peered":"2024-09-18T12:00:13.685939+0000","last_clean":"2024-09-18T12:00:13.685939+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:13.685939+0000","last_undegraded":"2024-09-18T12:00:13.685939+0000","last_fullsized":"2024-09-18T12:00:13.685939+0000","mapping_epoch":38,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28484818,"num_objects":6956,"num_object_clones":0,"num_object_copies":13912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12449,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12488","reported_seq":12567,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.081327+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:13.081327+0000","last_peered":"2024-09-18T12:00:13.081327+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:13.081327+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28501202,"num_objects":6960,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12486,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3428},{"shards":"0,2","objects":771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12538","reported_seq":12596,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.311828+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:13.311828+0000","last_peered":"2024-09-18T12:00:13.311828+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:13.311828+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28366034,"num_objects":6927,"num_object_clones":0,"num_object_copies":13854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12536,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":746},{"shards":"2","objects":3377}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12598","reported_seq":14952,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.339621+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:13.339621+0000","last_peered":"2024-09-18T12:00:13.339621+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:13.339621+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28262400,"num_objects":6900,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1778,"num_objects_misplaced":746,"num_objects_unfound":0,"num_objects_dirty":6900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9078,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1071,"num_bytes_recovered":4378624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":746},{"shards":"2","objects":4376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12530","reported_seq":12614,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:12.922800+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:12.922800+0000","last_peered":"2024-09-18T12:00:12.922800+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:12.922800+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28664425,"num_objects":6999,"num_object_clones":0,"num_object_copies":13998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12529,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3464},{"shards":"0,2","objects":770}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12557","reported_seq":19705,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:13.671453+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:13.671453+0000","last_peered":"2024-09-18T12:00:13.671453+0000","last_clean":"2024-09-18T12:00:13.671453+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:13.671453+0000","last_undegraded":"2024-09-18T12:00:13.671453+0000","last_fullsized":"2024-09-18T12:00:13.671453+0000","mapping_epoch":40,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28530491,"num_objects":6968,"num_object_clones":0,"num_object_copies":13936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12554,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12418","reported_seq":12501,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.063644+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:13.063644+0000","last_peered":"2024-09-18T12:00:13.063644+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:13.063644+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27772114,"num_objects":6782,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3189},{"shards":"0,2","objects":776}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12490","reported_seq":12577,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:13.091106+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:13.091106+0000","last_peered":"2024-09-18T12:00:13.091106+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:13.091106+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2400","ondisk_log_start":"30'2400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28764580,"num_objects":7026,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12487,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3503},{"shards":"0,2","objects":794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":455069772,"num_objects":111119,"num_object_clones":0,"num_object_copies":222238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33747,"num_objects_misplaced":746,"num_objects_unfound":0,"num_objects_dirty":111119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184096,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8099,"num_bytes_recovered":33063529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1052696576,"data_stored":1052625810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160903,"ondisk_log_size":160903,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051660,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634068,"kb_used_data":336204,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93737772,"statfs":{"total":96636764160,"available":95987478528,"internally_reserved":0,"allocated":344272896,"data_stored":344034780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.314},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.25600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":8.1750000000000007},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.28599999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855423,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617456,"kb_used_data":302948,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93754384,"statfs":{"total":96636764160,"available":96004489216,"internally_reserved":0,"allocated":310218752,"data_stored":309924829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.22},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.27000000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.36099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697136,"kb_used_data":391016,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93674704,"statfs":{"total":96636764160,"available":95922896896,"internally_reserved":0,"allocated":400400384,"data_stored":400162718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 11:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":0.98999999999999999,"15min":1.016},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0970000000000004,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.314},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.96999999999999997,"15min":1.097},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":5.7089999999999996,"5min":6.3520000000000003,"15min":6.3520000000000003},"last":8.3460000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.28299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":399192064,"data_stored":399188585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309780480,"data_stored":309720744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":343724032,"data_stored":343716481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:19.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:19.979 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:19.993 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-18T12:00:19.168941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":446500959,"num_objects":109029,"num_object_clones":0,"num_object_copies":218058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33789,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":109029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186190,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8197,"num_bytes_recovered":33460860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160798,"ondisk_log_size":160798,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1933052,"kb_used_data":1018716,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281182468,"statfs":{"total":289910292480,"available":287930847232,"internally_reserved":0,"allocated":1043165184,"data_stored":1042395479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,3,8,11,11,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25960448,"num_objects":-6338,"num_object_clones":0,"num_object_copies":-12676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":134,"num_objects_misplaced":-426,"num_objects_unfound":0,"num_objects_dirty":-6338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6338,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007964"},"pg_stats":[{"pgid":"2.f","version":"41'12697","reported_seq":12769,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.069019+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:18.069019+0000","last_peered":"2024-09-18T12:00:18.069019+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:18.069019+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28012544,"num_objects":6839,"num_object_clones":0,"num_object_copies":13678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12697,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3240},{"shards":"0,2","objects":670}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12621","reported_seq":12781,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.344453+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:18.344453+0000","last_peered":"2024-09-18T12:00:18.344453+0000","last_clean":"2024-09-18T12:00:18.344453+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:18.344453+0000","last_undegraded":"2024-09-18T12:00:18.344453+0000","last_fullsized":"2024-09-18T12:00:18.344453+0000","mapping_epoch":29,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27897856,"num_objects":6811,"num_object_clones":0,"num_object_copies":13622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9813,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12646","reported_seq":12831,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.273504+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:18.273504+0000","last_peered":"2024-09-18T12:00:18.273504+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:18.273504+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28307480,"num_objects":6912,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6912,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9504,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":670},{"shards":"2","objects":3376}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12535","reported_seq":12566,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.283969+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:18.283969+0000","last_peered":"2024-09-18T12:00:18.283969+0000","last_clean":"2024-09-18T12:00:18.283969+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:18.283969+0000","last_undegraded":"2024-09-18T12:00:18.283969+0000","last_fullsized":"2024-09-18T12:00:18.283969+0000","mapping_epoch":29,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27545600,"num_objects":6725,"num_object_clones":0,"num_object_copies":13450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12535,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12590","reported_seq":12783,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.060602+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:18.060602+0000","last_peered":"2024-09-18T12:00:18.060602+0000","last_clean":"2024-09-18T12:00:18.060602+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:18.060602+0000","last_undegraded":"2024-09-18T12:00:18.060602+0000","last_fullsized":"2024-09-18T12:00:18.060602+0000","mapping_epoch":29,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28086272,"num_objects":6857,"num_object_clones":0,"num_object_copies":13714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12609","reported_seq":12692,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.023985+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:18.023985+0000","last_peered":"2024-09-18T12:00:18.023985+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:18.023985+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27911378,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12607,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3289},{"shards":"0,2","objects":632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12581","reported_seq":12670,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.094628+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:18.094628+0000","last_peered":"2024-09-18T12:00:18.094628+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:18.094628+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27817170,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12579,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3234},{"shards":"0,2","objects":667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12809","reported_seq":13036,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.120464+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:18.120464+0000","last_peered":"2024-09-18T12:00:18.120464+0000","last_clean":"2024-09-18T12:00:18.120464+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:18.120464+0000","last_undegraded":"2024-09-18T12:00:18.120464+0000","last_fullsized":"2024-09-18T12:00:18.120464+0000","mapping_epoch":29,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27824128,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12569","reported_seq":19699,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.685143+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:18.685143+0000","last_peered":"2024-09-18T12:00:18.685143+0000","last_clean":"2024-09-18T12:00:18.685143+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:18.685143+0000","last_undegraded":"2024-09-18T12:00:18.685143+0000","last_fullsized":"2024-09-18T12:00:18.685143+0000","mapping_epoch":38,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28001490,"num_objects":6838,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12567,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12607","reported_seq":12687,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.060418+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:18.060418+0000","last_peered":"2024-09-18T12:00:18.060418+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:18.060418+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013778,"num_objects":6841,"num_object_clones":0,"num_object_copies":13682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12605,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3309},{"shards":"0,2","objects":652}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12672","reported_seq":12730,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.300171+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:18.300171+0000","last_peered":"2024-09-18T12:00:18.300171+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:18.300171+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27817170,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":612},{"shards":"2","objects":3243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12734","reported_seq":15282,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.333428+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:18.333428+0000","last_peered":"2024-09-18T12:00:18.333428+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:18.333428+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27705344,"num_objects":6764,"num_object_clones":0,"num_object_copies":13528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1820,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":6764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1168,"num_bytes_recovered":4775936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":607},{"shards":"2","objects":4337}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12667","reported_seq":12751,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.070310+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:18.070310+0000","last_peered":"2024-09-18T12:00:18.070310+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:18.070310+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28103273,"num_objects":6862,"num_object_clones":0,"num_object_copies":13724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3327},{"shards":"0,2","objects":633}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12690","reported_seq":19838,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.719913+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:18.719913+0000","last_peered":"2024-09-18T12:00:18.719913+0000","last_clean":"2024-09-18T12:00:18.719913+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:18.719913+0000","last_undegraded":"2024-09-18T12:00:18.719913+0000","last_fullsized":"2024-09-18T12:00:18.719913+0000","mapping_epoch":40,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27985723,"num_objects":6835,"num_object_clones":0,"num_object_copies":13670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12687,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12550","reported_seq":12630,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.075820+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:18.075820+0000","last_peered":"2024-09-18T12:00:18.075820+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:18.075820+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27231442,"num_objects":6650,"num_object_clones":0,"num_object_copies":13300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12548,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3057},{"shards":"0,2","objects":644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12618","reported_seq":12705,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.048132+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:18.048132+0000","last_peered":"2024-09-18T12:00:18.048132+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:18.048132+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28240292,"num_objects":6898,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12615,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3375},{"shards":"0,2","objects":666}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":446500940,"num_objects":109027,"num_object_clones":0,"num_object_copies":218054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33789,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":109027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186188,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8196,"num_bytes_recovered":33460841,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1040883712,"data_stored":1040812946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160795,"ondisk_log_size":160795,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051661,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634112,"kb_used_data":332084,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93737728,"statfs":{"total":96636764160,"available":95987433472,"internally_reserved":0,"allocated":340054016,"data_stored":339815900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,1,3,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.40000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855424,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603968,"kb_used_data":301944,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93767872,"statfs":{"total":96636764160,"available":96018300928,"internally_reserved":0,"allocated":309190656,"data_stored":308896733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":1.6859999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":1.774}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":1.1579999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.30199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694972,"kb_used_data":384688,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93676868,"statfs":{"total":96636764160,"available":95925112832,"internally_reserved":0,"allocated":393920512,"data_stored":393682846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":392708096,"data_stored":392704617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":308752384,"data_stored":308692648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339423232,"data_stored":339415681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:19.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:20.907 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:20.922 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:20.936 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-18T12:00:19.168941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":446500959,"num_objects":109029,"num_object_clones":0,"num_object_copies":218058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33789,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":109029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186190,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8197,"num_bytes_recovered":33460860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160798,"ondisk_log_size":160798,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1933052,"kb_used_data":1018716,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281182468,"statfs":{"total":289910292480,"available":287930847232,"internally_reserved":0,"allocated":1043165184,"data_stored":1042395479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,3,8,11,11,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25960448,"num_objects":-6338,"num_object_clones":0,"num_object_copies":-12676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":134,"num_objects_misplaced":-426,"num_objects_unfound":0,"num_objects_dirty":-6338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6338,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007964"},"pg_stats":[{"pgid":"2.f","version":"41'12697","reported_seq":12769,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.069019+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:18.069019+0000","last_peered":"2024-09-18T12:00:18.069019+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:18.069019+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28012544,"num_objects":6839,"num_object_clones":0,"num_object_copies":13678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12697,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3240},{"shards":"0,2","objects":670}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12621","reported_seq":12781,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.344453+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:18.344453+0000","last_peered":"2024-09-18T12:00:18.344453+0000","last_clean":"2024-09-18T12:00:18.344453+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:18.344453+0000","last_undegraded":"2024-09-18T12:00:18.344453+0000","last_fullsized":"2024-09-18T12:00:18.344453+0000","mapping_epoch":29,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27897856,"num_objects":6811,"num_object_clones":0,"num_object_copies":13622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9813,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12646","reported_seq":12831,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.273504+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:18.273504+0000","last_peered":"2024-09-18T12:00:18.273504+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:18.273504+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28307480,"num_objects":6912,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6912,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9504,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":670},{"shards":"2","objects":3376}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12535","reported_seq":12566,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.283969+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:18.283969+0000","last_peered":"2024-09-18T12:00:18.283969+0000","last_clean":"2024-09-18T12:00:18.283969+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:18.283969+0000","last_undegraded":"2024-09-18T12:00:18.283969+0000","last_fullsized":"2024-09-18T12:00:18.283969+0000","mapping_epoch":29,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27545600,"num_objects":6725,"num_object_clones":0,"num_object_copies":13450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12535,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12590","reported_seq":12783,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.060602+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:18.060602+0000","last_peered":"2024-09-18T12:00:18.060602+0000","last_clean":"2024-09-18T12:00:18.060602+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:18.060602+0000","last_undegraded":"2024-09-18T12:00:18.060602+0000","last_fullsized":"2024-09-18T12:00:18.060602+0000","mapping_epoch":29,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28086272,"num_objects":6857,"num_object_clones":0,"num_object_copies":13714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12609","reported_seq":12692,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.023985+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:18.023985+0000","last_peered":"2024-09-18T12:00:18.023985+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:18.023985+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27911378,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12607,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3289},{"shards":"0,2","objects":632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12581","reported_seq":12670,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.094628+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:18.094628+0000","last_peered":"2024-09-18T12:00:18.094628+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:18.094628+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27817170,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12579,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3234},{"shards":"0,2","objects":667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12809","reported_seq":13036,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.120464+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:18.120464+0000","last_peered":"2024-09-18T12:00:18.120464+0000","last_clean":"2024-09-18T12:00:18.120464+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:18.120464+0000","last_undegraded":"2024-09-18T12:00:18.120464+0000","last_fullsized":"2024-09-18T12:00:18.120464+0000","mapping_epoch":29,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27824128,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12569","reported_seq":19699,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.685143+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:18.685143+0000","last_peered":"2024-09-18T12:00:18.685143+0000","last_clean":"2024-09-18T12:00:18.685143+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:18.685143+0000","last_undegraded":"2024-09-18T12:00:18.685143+0000","last_fullsized":"2024-09-18T12:00:18.685143+0000","mapping_epoch":38,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28001490,"num_objects":6838,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12567,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12607","reported_seq":12687,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.060418+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:18.060418+0000","last_peered":"2024-09-18T12:00:18.060418+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:18.060418+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013778,"num_objects":6841,"num_object_clones":0,"num_object_copies":13682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12605,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3309},{"shards":"0,2","objects":652}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12672","reported_seq":12730,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.300171+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:18.300171+0000","last_peered":"2024-09-18T12:00:18.300171+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:18.300171+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27817170,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":612},{"shards":"2","objects":3243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12734","reported_seq":15282,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.333428+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:18.333428+0000","last_peered":"2024-09-18T12:00:18.333428+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:18.333428+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27705344,"num_objects":6764,"num_object_clones":0,"num_object_copies":13528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1820,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":6764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1168,"num_bytes_recovered":4775936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":607},{"shards":"2","objects":4337}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12667","reported_seq":12751,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.070310+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:18.070310+0000","last_peered":"2024-09-18T12:00:18.070310+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:18.070310+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28103273,"num_objects":6862,"num_object_clones":0,"num_object_copies":13724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3327},{"shards":"0,2","objects":633}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12690","reported_seq":19838,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.719913+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:18.719913+0000","last_peered":"2024-09-18T12:00:18.719913+0000","last_clean":"2024-09-18T12:00:18.719913+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:18.719913+0000","last_undegraded":"2024-09-18T12:00:18.719913+0000","last_fullsized":"2024-09-18T12:00:18.719913+0000","mapping_epoch":40,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27985723,"num_objects":6835,"num_object_clones":0,"num_object_copies":13670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12687,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12550","reported_seq":12630,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.075820+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:18.075820+0000","last_peered":"2024-09-18T12:00:18.075820+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:18.075820+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27231442,"num_objects":6650,"num_object_clones":0,"num_object_copies":13300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12548,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3057},{"shards":"0,2","objects":644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12618","reported_seq":12705,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.048132+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:18.048132+0000","last_peered":"2024-09-18T12:00:18.048132+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:18.048132+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28240292,"num_objects":6898,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12615,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3375},{"shards":"0,2","objects":666}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":446500940,"num_objects":109027,"num_object_clones":0,"num_object_copies":218054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33789,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":109027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186188,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8196,"num_bytes_recovered":33460841,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1040883712,"data_stored":1040812946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160795,"ondisk_log_size":160795,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051661,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634112,"kb_used_data":332084,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93737728,"statfs":{"total":96636764160,"available":95987433472,"internally_reserved":0,"allocated":340054016,"data_stored":339815900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,1,3,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.40000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855424,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603968,"kb_used_data":301944,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93767872,"statfs":{"total":96636764160,"available":96018300928,"internally_reserved":0,"allocated":309190656,"data_stored":308896733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":1.6859999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":1.774}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":1.1579999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.30199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694972,"kb_used_data":384688,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93676868,"statfs":{"total":96636764160,"available":95925112832,"internally_reserved":0,"allocated":393920512,"data_stored":393682846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":392708096,"data_stored":392704617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":308752384,"data_stored":308692648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339423232,"data_stored":339415681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:20.938 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:00:20.938 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'12697", 'reported_seq': 12769, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:18.069019+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:00:18.069019+0000', 'last_peered': '2024-09-18T12:00:18.069019+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:00:18.069019+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'2600", 'ondisk_log_start': "30'2600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28012544, 'num_objects': 6839, 'num_object_clones': 0, 'num_object_copies': 13678, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12697, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3240}, {'shards': '0,2', 'objects': 670}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:20.938 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:00:20.938 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'12646", 'reported_seq': 12831, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:18.273504+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:00:18.273504+0000', 'last_peered': '2024-09-18T12:00:18.273504+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:00:18.273504+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'2600", 'ondisk_log_start': "30'2600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28307480, 'num_objects': 6912, 'num_object_clones': 0, 'num_object_copies': 13824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6912, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9504, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 670}, {'shards': '2', 'objects': 3376}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:20.939 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:00:20.939 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'12609", 'reported_seq': 12692, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:18.023985+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:00:18.023985+0000', 'last_peered': '2024-09-18T12:00:18.023985+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:00:18.023985+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'2600", 'ondisk_log_start': "30'2600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27911378, 'num_objects': 6816, 'num_object_clones': 0, 'num_object_copies': 13632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6816, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12607, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3289}, {'shards': '0,2', 'objects': 632}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:20.939 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:00:20.939 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'12581", 'reported_seq': 12670, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:18.094628+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:00:18.094628+0000', 'last_peered': '2024-09-18T12:00:18.094628+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:00:18.094628+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'2500", 'ondisk_log_start': "30'2500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27817170, 'num_objects': 6793, 'num_object_clones': 0, 'num_object_copies': 13586, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12579, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3234}, {'shards': '0,2', 'objects': 667}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:20.939 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:00:20.939 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'12607", 'reported_seq': 12687, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:18.060418+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:00:18.060418+0000', 'last_peered': '2024-09-18T12:00:18.060418+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:00:18.060418+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'2600", 'ondisk_log_start': "30'2600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28013778, 'num_objects': 6841, 'num_object_clones': 0, 'num_object_copies': 13682, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6841, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12605, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3309}, {'shards': '0,2', 'objects': 652}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:20.940 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:00:20.940 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'12672", 'reported_seq': 12730, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:18.300171+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:00:18.300171+0000', 'last_peered': '2024-09-18T12:00:18.300171+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:00:18.300171+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'2600", 'ondisk_log_start': "30'2600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27817170, 'num_objects': 6793, 'num_object_clones': 0, 'num_object_copies': 13586, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12670, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 612}, {'shards': '2', 'objects': 3243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:20.940 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:00:20.940 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'12734", 'reported_seq': 15282, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:18.333428+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:00:18.333428+0000', 'last_peered': '2024-09-18T12:00:18.333428+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:00:18.333428+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'2700", 'ondisk_log_start': "30'2700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27705344, 'num_objects': 6764, 'num_object_clones': 0, 'num_object_copies': 13528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1820, 'num_objects_misplaced': 607, 'num_objects_unfound': 0, 'num_objects_dirty': 6764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9214, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1168, 'num_bytes_recovered': 4775936, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 607}, {'shards': '2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:20.940 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:00:20.940 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'12667", 'reported_seq': 12751, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:18.070310+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:00:18.070310+0000', 'last_peered': '2024-09-18T12:00:18.070310+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:00:18.070310+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'2600", 'ondisk_log_start': "30'2600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28103273, 'num_objects': 6862, 'num_object_clones': 0, 'num_object_copies': 13724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12666, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3327}, {'shards': '0,2', 'objects': 633}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:20.940 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:00:20.940 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'12550", 'reported_seq': 12630, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:18.075820+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:00:18.075820+0000', 'last_peered': '2024-09-18T12:00:18.075820+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:00:18.075820+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'2500", 'ondisk_log_start': "30'2500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27231442, 'num_objects': 6650, 'num_object_clones': 0, 'num_object_copies': 13300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6650, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12548, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3057}, {'shards': '0,2', 'objects': 644}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:20.940 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:00:20.941 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'12618", 'reported_seq': 12705, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:18.048132+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:00:18.048132+0000', 'last_peered': '2024-09-18T12:00:18.048132+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:00:18.048132+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'2600", 'ondisk_log_start': "30'2600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28240292, 'num_objects': 6898, 'num_object_clones': 0, 'num_object_copies': 13796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6898, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12615, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3375}, {'shards': '0,2', 'objects': 666}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:20.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:00:21.897 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:21.897 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33789/218058 objects degraded (15.495%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":421,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":109029,"data_bytes":446500959,"bytes_used":1979445248,"bytes_avail":287930847232,"bytes_total":289910292480,"degraded_objects":33789,"degraded_total":218058,"degraded_ratio":0.15495418650083922,"misplaced_objects":607,"misplaced_total":218058,"misplaced_ratio":0.0027836630621210865,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":64151,"recovering_keys_per_sec":0,"num_objects_recovered":188,"num_bytes_recovered":770048,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":351},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:00:21.930 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:00:21.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:22.693 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:22.714 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-18T12:00:21.169435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":446500959,"num_objects":109029,"num_object_clones":0,"num_object_copies":218058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33789,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":109029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186190,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8197,"num_bytes_recovered":33460860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160798,"ondisk_log_size":160798,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1933052,"kb_used_data":1018716,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281182468,"statfs":{"total":289910292480,"available":287930847232,"internally_reserved":0,"allocated":1043165184,"data_stored":1042395479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,3,8,11,11,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17305600,"num_objects":-4225,"num_object_clones":0,"num_object_copies":-8450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":104,"num_objects_misplaced":-292,"num_objects_unfound":0,"num_objects_dirty":-4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4225,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":770048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003608"},"pg_stats":[{"pgid":"2.f","version":"41'12697","reported_seq":12769,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.069019+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:18.069019+0000","last_peered":"2024-09-18T12:00:18.069019+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:18.069019+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28012544,"num_objects":6839,"num_object_clones":0,"num_object_copies":13678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12697,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3240},{"shards":"0,2","objects":670}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12621","reported_seq":12781,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.344453+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:18.344453+0000","last_peered":"2024-09-18T12:00:18.344453+0000","last_clean":"2024-09-18T12:00:18.344453+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:18.344453+0000","last_undegraded":"2024-09-18T12:00:18.344453+0000","last_fullsized":"2024-09-18T12:00:18.344453+0000","mapping_epoch":29,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27897856,"num_objects":6811,"num_object_clones":0,"num_object_copies":13622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9813,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12646","reported_seq":12831,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.273504+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:18.273504+0000","last_peered":"2024-09-18T12:00:18.273504+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:18.273504+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28307480,"num_objects":6912,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6912,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9504,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":670},{"shards":"2","objects":3376}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12535","reported_seq":12566,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.283969+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:18.283969+0000","last_peered":"2024-09-18T12:00:18.283969+0000","last_clean":"2024-09-18T12:00:18.283969+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:18.283969+0000","last_undegraded":"2024-09-18T12:00:18.283969+0000","last_fullsized":"2024-09-18T12:00:18.283969+0000","mapping_epoch":29,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27545600,"num_objects":6725,"num_object_clones":0,"num_object_copies":13450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12535,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12590","reported_seq":12783,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.060602+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:18.060602+0000","last_peered":"2024-09-18T12:00:18.060602+0000","last_clean":"2024-09-18T12:00:18.060602+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:18.060602+0000","last_undegraded":"2024-09-18T12:00:18.060602+0000","last_fullsized":"2024-09-18T12:00:18.060602+0000","mapping_epoch":29,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28086272,"num_objects":6857,"num_object_clones":0,"num_object_copies":13714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12609","reported_seq":12692,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.023985+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:18.023985+0000","last_peered":"2024-09-18T12:00:18.023985+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:18.023985+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27911378,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12607,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3289},{"shards":"0,2","objects":632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12581","reported_seq":12670,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.094628+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:18.094628+0000","last_peered":"2024-09-18T12:00:18.094628+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:18.094628+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27817170,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12579,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3234},{"shards":"0,2","objects":667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12809","reported_seq":13036,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.120464+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:18.120464+0000","last_peered":"2024-09-18T12:00:18.120464+0000","last_clean":"2024-09-18T12:00:18.120464+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:18.120464+0000","last_undegraded":"2024-09-18T12:00:18.120464+0000","last_fullsized":"2024-09-18T12:00:18.120464+0000","mapping_epoch":29,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27824128,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9126,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12569","reported_seq":19699,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.685143+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:18.685143+0000","last_peered":"2024-09-18T12:00:18.685143+0000","last_clean":"2024-09-18T12:00:18.685143+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:18.685143+0000","last_undegraded":"2024-09-18T12:00:18.685143+0000","last_fullsized":"2024-09-18T12:00:18.685143+0000","mapping_epoch":38,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28001490,"num_objects":6838,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12567,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12607","reported_seq":12687,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.060418+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:18.060418+0000","last_peered":"2024-09-18T12:00:18.060418+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:18.060418+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28013778,"num_objects":6841,"num_object_clones":0,"num_object_copies":13682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12605,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3309},{"shards":"0,2","objects":652}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12672","reported_seq":12730,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.300171+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:18.300171+0000","last_peered":"2024-09-18T12:00:18.300171+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:18.300171+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27817170,"num_objects":6793,"num_object_clones":0,"num_object_copies":13586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":612},{"shards":"2","objects":3243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12734","reported_seq":15282,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.333428+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:18.333428+0000","last_peered":"2024-09-18T12:00:18.333428+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:18.333428+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27705344,"num_objects":6764,"num_object_clones":0,"num_object_copies":13528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1820,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":6764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1168,"num_bytes_recovered":4775936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":607},{"shards":"2","objects":4337}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12667","reported_seq":12751,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.070310+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:18.070310+0000","last_peered":"2024-09-18T12:00:18.070310+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:18.070310+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28103273,"num_objects":6862,"num_object_clones":0,"num_object_copies":13724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3327},{"shards":"0,2","objects":633}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12690","reported_seq":19838,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:18.719913+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:18.719913+0000","last_peered":"2024-09-18T12:00:18.719913+0000","last_clean":"2024-09-18T12:00:18.719913+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:18.719913+0000","last_undegraded":"2024-09-18T12:00:18.719913+0000","last_fullsized":"2024-09-18T12:00:18.719913+0000","mapping_epoch":40,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27985723,"num_objects":6835,"num_object_clones":0,"num_object_copies":13670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12687,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12550","reported_seq":12630,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.075820+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:18.075820+0000","last_peered":"2024-09-18T12:00:18.075820+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:18.075820+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2500","ondisk_log_start":"30'2500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27231442,"num_objects":6650,"num_object_clones":0,"num_object_copies":13300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12548,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3057},{"shards":"0,2","objects":644}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12618","reported_seq":12705,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:18.048132+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:18.048132+0000","last_peered":"2024-09-18T12:00:18.048132+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:18.048132+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28240292,"num_objects":6898,"num_object_clones":0,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12615,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3375},{"shards":"0,2","objects":666}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":446500940,"num_objects":109027,"num_object_clones":0,"num_object_copies":218054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33789,"num_objects_misplaced":607,"num_objects_unfound":0,"num_objects_dirty":109027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186188,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8196,"num_bytes_recovered":33460841,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1040883712,"data_stored":1040812946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160795,"ondisk_log_size":160795,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051661,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634112,"kb_used_data":332084,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93737728,"statfs":{"total":96636764160,"available":95987433472,"internally_reserved":0,"allocated":340054016,"data_stored":339815900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,1,3,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.40000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855424,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603968,"kb_used_data":301944,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93767872,"statfs":{"total":96636764160,"available":96018300928,"internally_reserved":0,"allocated":309190656,"data_stored":308896733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":1.6859999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":1.774}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":1.1579999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.30199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694972,"kb_used_data":384688,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93676868,"statfs":{"total":96636764160,"available":95925112832,"internally_reserved":0,"allocated":393920512,"data_stored":393682846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":392708096,"data_stored":392704617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":308752384,"data_stored":308692648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339423232,"data_stored":339415681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:25.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:26.670 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:26.684 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-18T12:00:25.173867+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":437850207,"num_objects":106917,"num_object_clones":0,"num_object_copies":213834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33793,"num_objects_misplaced":486,"num_objects_unfound":0,"num_objects_dirty":106917,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188302,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8314,"num_bytes_recovered":33940092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1917676,"kb_used_data":1007508,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281197844,"statfs":{"total":289910292480,"available":287946592256,"internally_reserved":0,"allocated":1031688192,"data_stored":1030918487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,5,5,6,8,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21098496,"num_objects":-5151,"num_object_clones":0,"num_object_copies":-10302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":108,"num_objects_misplaced":-413,"num_objects_unfound":0,"num_objects_dirty":-5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5151,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":1249280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006370"},"pg_stats":[{"pgid":"2.f","version":"41'12832","reported_seq":12905,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.139237+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:23.139237+0000","last_peered":"2024-09-18T12:00:23.139237+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:23.139237+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27459584,"num_objects":6704,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12832,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3105},{"shards":"0,2","objects":535}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12749","reported_seq":12909,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:23.359351+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:23.359351+0000","last_peered":"2024-09-18T12:00:23.359351+0000","last_clean":"2024-09-18T12:00:23.359351+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:23.359351+0000","last_undegraded":"2024-09-18T12:00:23.359351+0000","last_fullsized":"2024-09-18T12:00:23.359351+0000","mapping_epoch":29,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27373568,"num_objects":6683,"num_object_clones":0,"num_object_copies":13366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9941,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12784","reported_seq":12972,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.292862+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:23.292862+0000","last_peered":"2024-09-18T12:00:23.292862+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:23.292862+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27742232,"num_objects":6774,"num_object_clones":0,"num_object_copies":13548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9642,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":532},{"shards":"2","objects":3238}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12661","reported_seq":12692,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:23.295023+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:23.295023+0000","last_peered":"2024-09-18T12:00:23.295023+0000","last_clean":"2024-09-18T12:00:23.295023+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:23.295023+0000","last_undegraded":"2024-09-18T12:00:23.295023+0000","last_fullsized":"2024-09-18T12:00:23.295023+0000","mapping_epoch":29,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27029504,"num_objects":6599,"num_object_clones":0,"num_object_copies":13198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12661,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12748","reported_seq":12941,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:23.067426+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:23.067426+0000","last_peered":"2024-09-18T12:00:23.067426+0000","last_clean":"2024-09-18T12:00:23.067426+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:23.067426+0000","last_undegraded":"2024-09-18T12:00:23.067426+0000","last_fullsized":"2024-09-18T12:00:23.067426+0000","mapping_epoch":29,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27439104,"num_objects":6699,"num_object_clones":0,"num_object_copies":13398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9913,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12746","reported_seq":12826,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.144512+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:23.144512+0000","last_peered":"2024-09-18T12:00:23.144512+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:23.144512+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27350226,"num_objects":6679,"num_object_clones":0,"num_object_copies":13358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12744,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3152},{"shards":"0,2","objects":495}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12731","reported_seq":12820,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.129301+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:23.129301+0000","last_peered":"2024-09-18T12:00:23.129301+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:23.129301+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27202770,"num_objects":6643,"num_object_clones":0,"num_object_copies":13286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12729,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3084},{"shards":"0,2","objects":517}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12942","reported_seq":13169,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:23.109787+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:23.109787+0000","last_peered":"2024-09-18T12:00:23.109787+0000","last_clean":"2024-09-18T12:00:23.109787+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:23.109787+0000","last_undegraded":"2024-09-18T12:00:23.109787+0000","last_fullsized":"2024-09-18T12:00:23.109787+0000","mapping_epoch":29,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27279360,"num_objects":6660,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9259,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12698","reported_seq":19828,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:23.780574+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:23.780574+0000","last_peered":"2024-09-18T12:00:23.780574+0000","last_clean":"2024-09-18T12:00:23.780574+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:23.780574+0000","last_undegraded":"2024-09-18T12:00:23.780574+0000","last_fullsized":"2024-09-18T12:00:23.780574+0000","mapping_epoch":38,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27473106,"num_objects":6709,"num_object_clones":0,"num_object_copies":13418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12696,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12730","reported_seq":12810,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.092143+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:23.092143+0000","last_peered":"2024-09-18T12:00:23.092143+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:23.092143+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27509970,"num_objects":6718,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12728,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3186},{"shards":"0,2","objects":529}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12802","reported_seq":12860,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.315721+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:23.315721+0000","last_peered":"2024-09-18T12:00:23.315721+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:23.315721+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27284690,"num_objects":6663,"num_object_clones":0,"num_object_copies":13326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12800,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":482},{"shards":"2","objects":3113}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12849","reported_seq":15631,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.352289+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:23.352289+0000","last_peered":"2024-09-18T12:00:23.352289+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:23.352289+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27234304,"num_objects":6649,"num_object_clones":0,"num_object_copies":13298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1824,"num_objects_misplaced":486,"num_objects_unfound":0,"num_objects_dirty":6649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9329,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1285,"num_bytes_recovered":5255168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":486},{"shards":"2","objects":4339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12800","reported_seq":12884,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.092700+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:23.092700+0000","last_peered":"2024-09-18T12:00:23.092700+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:23.092700+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27558505,"num_objects":6729,"num_object_clones":0,"num_object_copies":13458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12799,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3194},{"shards":"0,2","objects":500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12802","reported_seq":19950,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:23.767641+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:23.767641+0000","last_peered":"2024-09-18T12:00:23.767641+0000","last_clean":"2024-09-18T12:00:23.767641+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:23.767641+0000","last_undegraded":"2024-09-18T12:00:23.767641+0000","last_fullsized":"2024-09-18T12:00:23.767641+0000","mapping_epoch":40,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27526971,"num_objects":6723,"num_object_clones":0,"num_object_copies":13446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12799,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12692","reported_seq":12772,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.012031+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:23.012031+0000","last_peered":"2024-09-18T12:00:23.012031+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:23.012031+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26649810,"num_objects":6508,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12690,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2915},{"shards":"0,2","objects":502}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12741","reported_seq":12828,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:22.947152+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:22.947152+0000","last_peered":"2024-09-18T12:00:22.947152+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:22.947152+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27736484,"num_objects":6775,"num_object_clones":0,"num_object_copies":13550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12738,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3252},{"shards":"0,2","objects":543}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":437850188,"num_objects":106915,"num_object_clones":0,"num_object_copies":213830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33793,"num_objects_misplaced":486,"num_objects_unfound":0,"num_objects_dirty":106915,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188300,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8313,"num_bytes_recovered":33940073,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1029160960,"data_stored":1029090194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160807,"ondisk_log_size":160807,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051662,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634164,"kb_used_data":327980,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93737676,"statfs":{"total":96636764160,"available":95987380224,"internally_reserved":0,"allocated":335851520,"data_stored":335613404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.221},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":4.1929999999999996}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.39000000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855425,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607256,"kb_used_data":301072,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93764584,"statfs":{"total":96636764160,"available":96014934016,"internally_reserved":0,"allocated":308297728,"data_stored":308003805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":3.419}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.23200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676256,"kb_used_data":378456,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93695584,"statfs":{"total":96636764160,"available":95944278016,"internally_reserved":0,"allocated":387538944,"data_stored":387301278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.189}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":3.8969999999999998},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":386248704,"data_stored":386245225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307810304,"data_stored":307750568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":335101952,"data_stored":335094401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:26.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:27.409 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:27.417 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:27.438 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-18T12:00:27.174338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":437850207,"num_objects":106917,"num_object_clones":0,"num_object_copies":213834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33793,"num_objects_misplaced":486,"num_objects_unfound":0,"num_objects_dirty":106917,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188302,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8314,"num_bytes_recovered":33940092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1917676,"kb_used_data":1007508,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281197844,"statfs":{"total":289910292480,"available":287946592256,"internally_reserved":0,"allocated":1031688192,"data_stored":1030918487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,5,5,6,8,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17219584,"num_objects":-4204,"num_object_clones":0,"num_object_copies":-8408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":46,"num_objects_misplaced":-260,"num_objects_unfound":0,"num_objects_dirty":-4204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4204,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006360"},"pg_stats":[{"pgid":"2.f","version":"41'12832","reported_seq":12905,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.139237+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:23.139237+0000","last_peered":"2024-09-18T12:00:23.139237+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:23.139237+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27459584,"num_objects":6704,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12832,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3105},{"shards":"0,2","objects":535}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12749","reported_seq":12909,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:23.359351+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:23.359351+0000","last_peered":"2024-09-18T12:00:23.359351+0000","last_clean":"2024-09-18T12:00:23.359351+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:23.359351+0000","last_undegraded":"2024-09-18T12:00:23.359351+0000","last_fullsized":"2024-09-18T12:00:23.359351+0000","mapping_epoch":29,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27373568,"num_objects":6683,"num_object_clones":0,"num_object_copies":13366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9941,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12784","reported_seq":12972,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.292862+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:23.292862+0000","last_peered":"2024-09-18T12:00:23.292862+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:23.292862+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27742232,"num_objects":6774,"num_object_clones":0,"num_object_copies":13548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9642,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":532},{"shards":"2","objects":3238}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12661","reported_seq":12692,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:23.295023+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:23.295023+0000","last_peered":"2024-09-18T12:00:23.295023+0000","last_clean":"2024-09-18T12:00:23.295023+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:23.295023+0000","last_undegraded":"2024-09-18T12:00:23.295023+0000","last_fullsized":"2024-09-18T12:00:23.295023+0000","mapping_epoch":29,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27029504,"num_objects":6599,"num_object_clones":0,"num_object_copies":13198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12661,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12748","reported_seq":12941,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:23.067426+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:23.067426+0000","last_peered":"2024-09-18T12:00:23.067426+0000","last_clean":"2024-09-18T12:00:23.067426+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:23.067426+0000","last_undegraded":"2024-09-18T12:00:23.067426+0000","last_fullsized":"2024-09-18T12:00:23.067426+0000","mapping_epoch":29,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27439104,"num_objects":6699,"num_object_clones":0,"num_object_copies":13398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9913,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12746","reported_seq":12826,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.144512+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:23.144512+0000","last_peered":"2024-09-18T12:00:23.144512+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:23.144512+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27350226,"num_objects":6679,"num_object_clones":0,"num_object_copies":13358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12744,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3152},{"shards":"0,2","objects":495}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12731","reported_seq":12820,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.129301+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:23.129301+0000","last_peered":"2024-09-18T12:00:23.129301+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:23.129301+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27202770,"num_objects":6643,"num_object_clones":0,"num_object_copies":13286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12729,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3084},{"shards":"0,2","objects":517}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'12942","reported_seq":13169,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:23.109787+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:23.109787+0000","last_peered":"2024-09-18T12:00:23.109787+0000","last_clean":"2024-09-18T12:00:23.109787+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:23.109787+0000","last_undegraded":"2024-09-18T12:00:23.109787+0000","last_fullsized":"2024-09-18T12:00:23.109787+0000","mapping_epoch":29,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27279360,"num_objects":6660,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9259,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12698","reported_seq":19828,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:23.780574+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:23.780574+0000","last_peered":"2024-09-18T12:00:23.780574+0000","last_clean":"2024-09-18T12:00:23.780574+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:23.780574+0000","last_undegraded":"2024-09-18T12:00:23.780574+0000","last_fullsized":"2024-09-18T12:00:23.780574+0000","mapping_epoch":38,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27473106,"num_objects":6709,"num_object_clones":0,"num_object_copies":13418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12696,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12730","reported_seq":12810,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.092143+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:23.092143+0000","last_peered":"2024-09-18T12:00:23.092143+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:23.092143+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27509970,"num_objects":6718,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12728,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3186},{"shards":"0,2","objects":529}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12802","reported_seq":12860,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.315721+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:23.315721+0000","last_peered":"2024-09-18T12:00:23.315721+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:23.315721+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27284690,"num_objects":6663,"num_object_clones":0,"num_object_copies":13326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12800,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":482},{"shards":"2","objects":3113}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12849","reported_seq":15631,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.352289+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:23.352289+0000","last_peered":"2024-09-18T12:00:23.352289+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:23.352289+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27234304,"num_objects":6649,"num_object_clones":0,"num_object_copies":13298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1824,"num_objects_misplaced":486,"num_objects_unfound":0,"num_objects_dirty":6649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9329,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1285,"num_bytes_recovered":5255168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":486},{"shards":"2","objects":4339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12800","reported_seq":12884,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.092700+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:23.092700+0000","last_peered":"2024-09-18T12:00:23.092700+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:23.092700+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27558505,"num_objects":6729,"num_object_clones":0,"num_object_copies":13458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12799,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3194},{"shards":"0,2","objects":500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12802","reported_seq":19950,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:23.767641+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:23.767641+0000","last_peered":"2024-09-18T12:00:23.767641+0000","last_clean":"2024-09-18T12:00:23.767641+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:23.767641+0000","last_undegraded":"2024-09-18T12:00:23.767641+0000","last_fullsized":"2024-09-18T12:00:23.767641+0000","mapping_epoch":40,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27526971,"num_objects":6723,"num_object_clones":0,"num_object_copies":13446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12799,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12692","reported_seq":12772,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:23.012031+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:23.012031+0000","last_peered":"2024-09-18T12:00:23.012031+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:23.012031+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2600","ondisk_log_start":"30'2600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26649810,"num_objects":6508,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12690,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2915},{"shards":"0,2","objects":502}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12741","reported_seq":12828,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:22.947152+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:22.947152+0000","last_peered":"2024-09-18T12:00:22.947152+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:22.947152+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27736484,"num_objects":6775,"num_object_clones":0,"num_object_copies":13550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12738,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3252},{"shards":"0,2","objects":543}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":437850188,"num_objects":106915,"num_object_clones":0,"num_object_copies":213830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33793,"num_objects_misplaced":486,"num_objects_unfound":0,"num_objects_dirty":106915,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188300,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8313,"num_bytes_recovered":33940073,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1029160960,"data_stored":1029090194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160807,"ondisk_log_size":160807,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051662,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634164,"kb_used_data":327980,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93737676,"statfs":{"total":96636764160,"available":95987380224,"internally_reserved":0,"allocated":335851520,"data_stored":335613404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 11:59:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.68500000000000005,"15min":0.47999999999999998},"min":{"1min":0.184,"5min":0.153,"15min":0.153},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.221},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2090000000000001,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.4450000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":4.1929999999999996}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.39000000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855425,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607256,"kb_used_data":301072,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93764584,"statfs":{"total":96636764160,"available":96014934016,"internally_reserved":0,"allocated":308297728,"data_stored":308003805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":3.419}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.23200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676256,"kb_used_data":378456,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93695584,"statfs":{"total":96636764160,"available":95944278016,"internally_reserved":0,"allocated":387538944,"data_stored":387301278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.189}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":3.8969999999999998},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":386248704,"data_stored":386245225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307810304,"data_stored":307750568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":335101952,"data_stored":335094401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:27.439 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:00:27.439 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'12832", 'reported_seq': 12905, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:23.139237+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:00:23.139237+0000', 'last_peered': '2024-09-18T12:00:23.139237+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:00:23.139237+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'2800", 'ondisk_log_start': "30'2800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27459584, 'num_objects': 6704, 'num_object_clones': 0, 'num_object_copies': 13408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6704, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12832, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3105}, {'shards': '0,2', 'objects': 535}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:27.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:00:27.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'12784", 'reported_seq': 12972, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:23.292862+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:00:23.292862+0000', 'last_peered': '2024-09-18T12:00:23.292862+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:00:23.292862+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'2700", 'ondisk_log_start': "30'2700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27742232, 'num_objects': 6774, 'num_object_clones': 0, 'num_object_copies': 13548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6774, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9642, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 532}, {'shards': '2', 'objects': 3238}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:27.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:00:27.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'12746", 'reported_seq': 12826, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:23.144512+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:00:23.144512+0000', 'last_peered': '2024-09-18T12:00:23.144512+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:00:23.144512+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'2700", 'ondisk_log_start': "30'2700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27350226, 'num_objects': 6679, 'num_object_clones': 0, 'num_object_copies': 13358, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6679, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12744, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3152}, {'shards': '0,2', 'objects': 495}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:27.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:00:27.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'12731", 'reported_seq': 12820, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:23.129301+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:00:23.129301+0000', 'last_peered': '2024-09-18T12:00:23.129301+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:00:23.129301+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'2700", 'ondisk_log_start': "30'2700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27202770, 'num_objects': 6643, 'num_object_clones': 0, 'num_object_copies': 13286, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6643, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12729, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3084}, {'shards': '0,2', 'objects': 517}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:27.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:00:27.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'12730", 'reported_seq': 12810, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:23.092143+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:00:23.092143+0000', 'last_peered': '2024-09-18T12:00:23.092143+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:00:23.092143+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'2700", 'ondisk_log_start': "30'2700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27509970, 'num_objects': 6718, 'num_object_clones': 0, 'num_object_copies': 13436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12728, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3186}, {'shards': '0,2', 'objects': 529}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:27.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:00:27.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'12802", 'reported_seq': 12860, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:23.315721+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:00:23.315721+0000', 'last_peered': '2024-09-18T12:00:23.315721+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:00:23.315721+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'2800", 'ondisk_log_start': "30'2800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27284690, 'num_objects': 6663, 'num_object_clones': 0, 'num_object_copies': 13326, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12800, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 482}, {'shards': '2', 'objects': 3113}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:27.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:00:27.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'12849", 'reported_seq': 15631, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:23.352289+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:00:23.352289+0000', 'last_peered': '2024-09-18T12:00:23.352289+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:00:23.352289+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'2800", 'ondisk_log_start': "30'2800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27234304, 'num_objects': 6649, 'num_object_clones': 0, 'num_object_copies': 13298, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1824, 'num_objects_misplaced': 486, 'num_objects_unfound': 0, 'num_objects_dirty': 6649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9329, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1285, 'num_bytes_recovered': 5255168, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 486}, {'shards': '2', 'objects': 4339}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:27.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:00:27.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'12800", 'reported_seq': 12884, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:23.092700+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:00:23.092700+0000', 'last_peered': '2024-09-18T12:00:23.092700+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:00:23.092700+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'2700", 'ondisk_log_start': "30'2700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27558505, 'num_objects': 6729, 'num_object_clones': 0, 'num_object_copies': 13458, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12799, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3194}, {'shards': '0,2', 'objects': 500}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:27.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:00:27.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'12692", 'reported_seq': 12772, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:23.012031+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:00:23.012031+0000', 'last_peered': '2024-09-18T12:00:23.012031+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:00:23.012031+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'2600", 'ondisk_log_start': "30'2600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26649810, 'num_objects': 6508, 'num_object_clones': 0, 'num_object_copies': 13016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12690, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2915}, {'shards': '0,2', 'objects': 502}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:27.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:00:27.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'12741", 'reported_seq': 12828, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:22.947152+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:00:22.947152+0000', 'last_peered': '2024-09-18T12:00:22.947152+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:00:22.947152+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'2700", 'ondisk_log_start': "30'2700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27736484, 'num_objects': 6775, 'num_object_clones': 0, 'num_object_copies': 13550, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12738, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3252}, {'shards': '0,2', 'objects': 543}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:27.441 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:00:28.439 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:28.440 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33793/213834 objects degraded (15.803%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":428,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":106917,"data_bytes":437850207,"bytes_used":1963700224,"bytes_avail":287946592256,"bytes_total":289910292480,"degraded_objects":33793,"degraded_total":213834,"degraded_ratio":0.15803380192111638,"misplaced_objects":486,"misplaced_total":213834,"misplaced_ratio":0.0022727910435197396,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":73006,"recovering_keys_per_sec":0,"num_objects_recovered":214,"num_bytes_recovered":876544,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":350},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:00:28.496 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:00:28.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:29.297 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:29.309 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:29.315 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-18T12:00:29.174809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":429064287,"num_objects":104772,"num_object_clones":0,"num_object_copies":209544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33812,"num_objects_misplaced":352,"num_objects_unfound":0,"num_objects_dirty":104772,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190447,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8429,"num_bytes_recovered":34411132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160955,"ondisk_log_size":160955,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1905872,"kb_used_data":995676,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281209648,"statfs":{"total":289910292480,"available":287958679552,"internally_reserved":0,"allocated":1019572224,"data_stored":1018802519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,3,12,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26005504,"num_objects":-6349,"num_object_clones":0,"num_object_copies":-12698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":-394,"num_objects_unfound":0,"num_objects_dirty":-6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":329,"num_bytes_recovered":1347584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006345"},"pg_stats":[{"pgid":"2.f","version":"41'12962","reported_seq":13035,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.161018+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:28.161018+0000","last_peered":"2024-09-18T12:00:28.161018+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:28.161018+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26927104,"num_objects":6574,"num_object_clones":0,"num_object_copies":13148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12962,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2975},{"shards":"0,2","objects":405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12894","reported_seq":13054,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.304543+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:28.304543+0000","last_peered":"2024-09-18T12:00:28.304543+0000","last_clean":"2024-09-18T12:00:28.304543+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:28.304543+0000","last_undegraded":"2024-09-18T12:00:28.304543+0000","last_fullsized":"2024-09-18T12:00:28.304543+0000","mapping_epoch":29,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26779648,"num_objects":6538,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10086,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12939","reported_seq":13125,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.390408+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:28.390408+0000","last_peered":"2024-09-18T12:00:28.390408+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:28.390408+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27107352,"num_objects":6619,"num_object_clones":0,"num_object_copies":13238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6619,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9797,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":377},{"shards":"2","objects":3083}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12780","reported_seq":12811,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.361481+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:28.361481+0000","last_peered":"2024-09-18T12:00:28.361481+0000","last_clean":"2024-09-18T12:00:28.361481+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:28.361481+0000","last_undegraded":"2024-09-18T12:00:28.361481+0000","last_fullsized":"2024-09-18T12:00:28.361481+0000","mapping_epoch":29,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":6480,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12780,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12876","reported_seq":13068,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.167029+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:28.167029+0000","last_peered":"2024-09-18T12:00:28.167029+0000","last_clean":"2024-09-18T12:00:28.167029+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:28.167029+0000","last_undegraded":"2024-09-18T12:00:28.167029+0000","last_fullsized":"2024-09-18T12:00:28.167029+0000","mapping_epoch":29,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26914816,"num_objects":6571,"num_object_clones":0,"num_object_copies":13142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10041,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12871","reported_seq":12954,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.113100+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:28.113100+0000","last_peered":"2024-09-18T12:00:28.113100+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:28.113100+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26838226,"num_objects":6554,"num_object_clones":0,"num_object_copies":13108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12869,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3027},{"shards":"0,2","objects":370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12856","reported_seq":12945,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.124592+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:28.124592+0000","last_peered":"2024-09-18T12:00:28.124592+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:28.124592+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26690770,"num_objects":6518,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12854,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2959},{"shards":"0,2","objects":392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13084","reported_seq":13311,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.087544+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:28.087544+0000","last_peered":"2024-09-18T12:00:28.087544+0000","last_clean":"2024-09-18T12:00:28.087544+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:28.087544+0000","last_undegraded":"2024-09-18T12:00:28.087544+0000","last_fullsized":"2024-09-18T12:00:28.087544+0000","mapping_epoch":29,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26697728,"num_objects":6518,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9401,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12827","reported_seq":19957,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.838371+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:28.838371+0000","last_peered":"2024-09-18T12:00:28.838371+0000","last_clean":"2024-09-18T12:00:28.838371+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:28.838371+0000","last_undegraded":"2024-09-18T12:00:28.838371+0000","last_fullsized":"2024-09-18T12:00:28.838371+0000","mapping_epoch":38,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944722,"num_objects":6580,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12825,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12867","reported_seq":12947,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.111813+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:28.111813+0000","last_peered":"2024-09-18T12:00:28.111813+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:28.111813+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26948818,"num_objects":6581,"num_object_clones":0,"num_object_copies":13162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12865,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3049},{"shards":"0,2","objects":392}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12940","reported_seq":12998,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.272886+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:28.272886+0000","last_peered":"2024-09-18T12:00:28.272886+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:28.272886+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719442,"num_objects":6525,"num_object_clones":0,"num_object_copies":13050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12938,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":344},{"shards":"2","objects":2975}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12984","reported_seq":15994,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.342593+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:28.342593+0000","last_peered":"2024-09-18T12:00:28.342593+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:28.342593+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26681344,"num_objects":6514,"num_object_clones":0,"num_object_copies":13028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1843,"num_objects_misplaced":352,"num_objects_unfound":0,"num_objects_dirty":6514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9464,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1400,"num_bytes_recovered":5726208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":352},{"shards":"2","objects":4319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12931","reported_seq":13015,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:27.955646+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:27.955646+0000","last_peered":"2024-09-18T12:00:27.955646+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:27.955646+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27021929,"num_objects":6598,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12930,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3063},{"shards":"0,2","objects":369}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12934","reported_seq":20082,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.708130+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:28.708130+0000","last_peered":"2024-09-18T12:00:28.708130+0000","last_clean":"2024-09-18T12:00:28.708130+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:28.708130+0000","last_undegraded":"2024-09-18T12:00:28.708130+0000","last_fullsized":"2024-09-18T12:00:28.708130+0000","mapping_epoch":40,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26986299,"num_objects":6591,"num_object_clones":0,"num_object_copies":13182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12931,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12827","reported_seq":12910,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.089899+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:28.089899+0000","last_peered":"2024-09-18T12:00:28.089899+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:28.089899+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26096850,"num_objects":6373,"num_object_clones":0,"num_object_copies":12746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12825,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2780},{"shards":"0,2","objects":367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12880","reported_seq":12964,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.157128+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:28.157128+0000","last_peered":"2024-09-18T12:00:28.157128+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:28.157128+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27167140,"num_objects":6636,"num_object_clones":0,"num_object_copies":13272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12877,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3113},{"shards":"0,2","objects":404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":429064268,"num_objects":104770,"num_object_clones":0,"num_object_copies":209540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33812,"num_objects_misplaced":352,"num_objects_unfound":0,"num_objects_dirty":104770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190445,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8428,"num_bytes_recovered":34411113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1017090048,"data_stored":1017019282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160952,"ondisk_log_size":160952,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051663,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633980,"kb_used_data":323620,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93737860,"statfs":{"total":96636764160,"available":95987568640,"internally_reserved":0,"allocated":331386880,"data_stored":331148764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.26200000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":1.7210000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855426,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597888,"kb_used_data":300020,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93773952,"statfs":{"total":96636764160,"available":96024526848,"internally_reserved":0,"allocated":307220480,"data_stored":306926557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":3.4860000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.7690000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.32300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674004,"kb_used_data":372036,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93697836,"statfs":{"total":96636764160,"available":95946584064,"internally_reserved":0,"allocated":380964864,"data_stored":380727198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.1500000000000004},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.32600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.217},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":4.1890000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379662336,"data_stored":379658857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306720768,"data_stored":306661032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330706944,"data_stored":330699393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:32.317 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:33.170 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:33.199 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:33.213 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-18T12:00:31.175277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":429064287,"num_objects":104772,"num_object_clones":0,"num_object_copies":209544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33812,"num_objects_misplaced":352,"num_objects_unfound":0,"num_objects_dirty":104772,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190447,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8429,"num_bytes_recovered":34411132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160955,"ondisk_log_size":160955,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1905872,"kb_used_data":995676,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281209648,"statfs":{"total":289910292480,"available":287958679552,"internally_reserved":0,"allocated":1019572224,"data_stored":1018802519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,3,12,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17436672,"num_objects":-4257,"num_object_clones":0,"num_object_copies":-8514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23,"num_objects_misplaced":-255,"num_objects_unfound":0,"num_objects_dirty":-4257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4257,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006336"},"pg_stats":[{"pgid":"2.f","version":"41'12962","reported_seq":13035,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.161018+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:28.161018+0000","last_peered":"2024-09-18T12:00:28.161018+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:28.161018+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26927104,"num_objects":6574,"num_object_clones":0,"num_object_copies":13148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12962,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2975},{"shards":"0,2","objects":405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12894","reported_seq":13054,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.304543+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:28.304543+0000","last_peered":"2024-09-18T12:00:28.304543+0000","last_clean":"2024-09-18T12:00:28.304543+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:28.304543+0000","last_undegraded":"2024-09-18T12:00:28.304543+0000","last_fullsized":"2024-09-18T12:00:28.304543+0000","mapping_epoch":29,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26779648,"num_objects":6538,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10086,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12939","reported_seq":13125,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.390408+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:28.390408+0000","last_peered":"2024-09-18T12:00:28.390408+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:28.390408+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27107352,"num_objects":6619,"num_object_clones":0,"num_object_copies":13238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6619,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9797,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":377},{"shards":"2","objects":3083}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12780","reported_seq":12811,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.361481+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:28.361481+0000","last_peered":"2024-09-18T12:00:28.361481+0000","last_clean":"2024-09-18T12:00:28.361481+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:28.361481+0000","last_undegraded":"2024-09-18T12:00:28.361481+0000","last_fullsized":"2024-09-18T12:00:28.361481+0000","mapping_epoch":29,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":6480,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12780,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12876","reported_seq":13068,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.167029+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:28.167029+0000","last_peered":"2024-09-18T12:00:28.167029+0000","last_clean":"2024-09-18T12:00:28.167029+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:28.167029+0000","last_undegraded":"2024-09-18T12:00:28.167029+0000","last_fullsized":"2024-09-18T12:00:28.167029+0000","mapping_epoch":29,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26914816,"num_objects":6571,"num_object_clones":0,"num_object_copies":13142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10041,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12871","reported_seq":12954,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.113100+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:28.113100+0000","last_peered":"2024-09-18T12:00:28.113100+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:28.113100+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26838226,"num_objects":6554,"num_object_clones":0,"num_object_copies":13108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12869,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3027},{"shards":"0,2","objects":370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12856","reported_seq":12945,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.124592+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:28.124592+0000","last_peered":"2024-09-18T12:00:28.124592+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:28.124592+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26690770,"num_objects":6518,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12854,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2959},{"shards":"0,2","objects":392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13084","reported_seq":13311,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.087544+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:28.087544+0000","last_peered":"2024-09-18T12:00:28.087544+0000","last_clean":"2024-09-18T12:00:28.087544+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:28.087544+0000","last_undegraded":"2024-09-18T12:00:28.087544+0000","last_fullsized":"2024-09-18T12:00:28.087544+0000","mapping_epoch":29,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26697728,"num_objects":6518,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9401,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12827","reported_seq":19957,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.838371+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:28.838371+0000","last_peered":"2024-09-18T12:00:28.838371+0000","last_clean":"2024-09-18T12:00:28.838371+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:28.838371+0000","last_undegraded":"2024-09-18T12:00:28.838371+0000","last_fullsized":"2024-09-18T12:00:28.838371+0000","mapping_epoch":38,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944722,"num_objects":6580,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12825,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12867","reported_seq":12947,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.111813+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:28.111813+0000","last_peered":"2024-09-18T12:00:28.111813+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:28.111813+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26948818,"num_objects":6581,"num_object_clones":0,"num_object_copies":13162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12865,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3049},{"shards":"0,2","objects":392}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12940","reported_seq":12998,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.272886+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:28.272886+0000","last_peered":"2024-09-18T12:00:28.272886+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:28.272886+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719442,"num_objects":6525,"num_object_clones":0,"num_object_copies":13050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12938,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":344},{"shards":"2","objects":2975}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12984","reported_seq":15994,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.342593+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:28.342593+0000","last_peered":"2024-09-18T12:00:28.342593+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:28.342593+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26681344,"num_objects":6514,"num_object_clones":0,"num_object_copies":13028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1843,"num_objects_misplaced":352,"num_objects_unfound":0,"num_objects_dirty":6514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9464,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1400,"num_bytes_recovered":5726208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":352},{"shards":"2","objects":4319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12931","reported_seq":13015,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:27.955646+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:27.955646+0000","last_peered":"2024-09-18T12:00:27.955646+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:27.955646+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27021929,"num_objects":6598,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12930,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3063},{"shards":"0,2","objects":369}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12934","reported_seq":20082,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.708130+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:28.708130+0000","last_peered":"2024-09-18T12:00:28.708130+0000","last_clean":"2024-09-18T12:00:28.708130+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:28.708130+0000","last_undegraded":"2024-09-18T12:00:28.708130+0000","last_fullsized":"2024-09-18T12:00:28.708130+0000","mapping_epoch":40,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26986299,"num_objects":6591,"num_object_clones":0,"num_object_copies":13182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12931,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12827","reported_seq":12910,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.089899+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:28.089899+0000","last_peered":"2024-09-18T12:00:28.089899+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:28.089899+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26096850,"num_objects":6373,"num_object_clones":0,"num_object_copies":12746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12825,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2780},{"shards":"0,2","objects":367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12880","reported_seq":12964,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.157128+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:28.157128+0000","last_peered":"2024-09-18T12:00:28.157128+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:28.157128+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27167140,"num_objects":6636,"num_object_clones":0,"num_object_copies":13272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12877,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3113},{"shards":"0,2","objects":404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":429064268,"num_objects":104770,"num_object_clones":0,"num_object_copies":209540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33812,"num_objects_misplaced":352,"num_objects_unfound":0,"num_objects_dirty":104770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190445,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8428,"num_bytes_recovered":34411113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1017090048,"data_stored":1017019282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160952,"ondisk_log_size":160952,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051663,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633980,"kb_used_data":323620,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93737860,"statfs":{"total":96636764160,"available":95987568640,"internally_reserved":0,"allocated":331386880,"data_stored":331148764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.26200000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":1.7210000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855426,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597888,"kb_used_data":300020,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93773952,"statfs":{"total":96636764160,"available":96024526848,"internally_reserved":0,"allocated":307220480,"data_stored":306926557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":3.4860000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.7690000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.32300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674004,"kb_used_data":372036,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93697836,"statfs":{"total":96636764160,"available":95946584064,"internally_reserved":0,"allocated":380964864,"data_stored":380727198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.1500000000000004},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.32600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.217},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":4.1890000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379662336,"data_stored":379658857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306720768,"data_stored":306661032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330706944,"data_stored":330699393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:33.215 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:33.953 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:33.967 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-18T12:00:33.175708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":429064287,"num_objects":104772,"num_object_clones":0,"num_object_copies":209544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33812,"num_objects_misplaced":352,"num_objects_unfound":0,"num_objects_dirty":104772,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190447,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8429,"num_bytes_recovered":34411132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160955,"ondisk_log_size":160955,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1905872,"kb_used_data":995676,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281209648,"statfs":{"total":289910292480,"available":287958679552,"internally_reserved":0,"allocated":1019572224,"data_stored":1018802519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,3,12,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17436672,"num_objects":-4257,"num_object_clones":0,"num_object_copies":-8514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23,"num_objects_misplaced":-255,"num_objects_unfound":0,"num_objects_dirty":-4257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4257,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006272"},"pg_stats":[{"pgid":"2.f","version":"41'12962","reported_seq":13035,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.161018+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:28.161018+0000","last_peered":"2024-09-18T12:00:28.161018+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:28.161018+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26927104,"num_objects":6574,"num_object_clones":0,"num_object_copies":13148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12962,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2975},{"shards":"0,2","objects":405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'12894","reported_seq":13054,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.304543+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:28.304543+0000","last_peered":"2024-09-18T12:00:28.304543+0000","last_clean":"2024-09-18T12:00:28.304543+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:28.304543+0000","last_undegraded":"2024-09-18T12:00:28.304543+0000","last_fullsized":"2024-09-18T12:00:28.304543+0000","mapping_epoch":29,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26779648,"num_objects":6538,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10086,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'12939","reported_seq":13125,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.390408+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:28.390408+0000","last_peered":"2024-09-18T12:00:28.390408+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:28.390408+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27107352,"num_objects":6619,"num_object_clones":0,"num_object_copies":13238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6619,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9797,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":377},{"shards":"2","objects":3083}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12780","reported_seq":12811,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.361481+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:28.361481+0000","last_peered":"2024-09-18T12:00:28.361481+0000","last_clean":"2024-09-18T12:00:28.361481+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:28.361481+0000","last_undegraded":"2024-09-18T12:00:28.361481+0000","last_fullsized":"2024-09-18T12:00:28.361481+0000","mapping_epoch":29,"log_start":"30'2700","ondisk_log_start":"30'2700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":6480,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12780,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'12876","reported_seq":13068,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.167029+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:28.167029+0000","last_peered":"2024-09-18T12:00:28.167029+0000","last_clean":"2024-09-18T12:00:28.167029+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:28.167029+0000","last_undegraded":"2024-09-18T12:00:28.167029+0000","last_fullsized":"2024-09-18T12:00:28.167029+0000","mapping_epoch":29,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26914816,"num_objects":6571,"num_object_clones":0,"num_object_copies":13142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10041,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12871","reported_seq":12954,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.113100+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:28.113100+0000","last_peered":"2024-09-18T12:00:28.113100+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:28.113100+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26838226,"num_objects":6554,"num_object_clones":0,"num_object_copies":13108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12869,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3027},{"shards":"0,2","objects":370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12856","reported_seq":12945,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.124592+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:28.124592+0000","last_peered":"2024-09-18T12:00:28.124592+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:28.124592+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26690770,"num_objects":6518,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12854,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2959},{"shards":"0,2","objects":392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13084","reported_seq":13311,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.087544+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:28.087544+0000","last_peered":"2024-09-18T12:00:28.087544+0000","last_clean":"2024-09-18T12:00:28.087544+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:28.087544+0000","last_undegraded":"2024-09-18T12:00:28.087544+0000","last_fullsized":"2024-09-18T12:00:28.087544+0000","mapping_epoch":29,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26697728,"num_objects":6518,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9401,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12827","reported_seq":19957,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.838371+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:28.838371+0000","last_peered":"2024-09-18T12:00:28.838371+0000","last_clean":"2024-09-18T12:00:28.838371+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:28.838371+0000","last_undegraded":"2024-09-18T12:00:28.838371+0000","last_fullsized":"2024-09-18T12:00:28.838371+0000","mapping_epoch":38,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26944722,"num_objects":6580,"num_object_clones":0,"num_object_copies":13160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12825,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'12867","reported_seq":12947,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.111813+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:28.111813+0000","last_peered":"2024-09-18T12:00:28.111813+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:28.111813+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26948818,"num_objects":6581,"num_object_clones":0,"num_object_copies":13162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12865,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3049},{"shards":"0,2","objects":392}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'12940","reported_seq":12998,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.272886+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:28.272886+0000","last_peered":"2024-09-18T12:00:28.272886+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:28.272886+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719442,"num_objects":6525,"num_object_clones":0,"num_object_copies":13050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12938,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":344},{"shards":"2","objects":2975}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'12984","reported_seq":15994,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.342593+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:28.342593+0000","last_peered":"2024-09-18T12:00:28.342593+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:28.342593+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26681344,"num_objects":6514,"num_object_clones":0,"num_object_copies":13028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1843,"num_objects_misplaced":352,"num_objects_unfound":0,"num_objects_dirty":6514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9464,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1400,"num_bytes_recovered":5726208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":352},{"shards":"2","objects":4319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'12931","reported_seq":13015,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:27.955646+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:27.955646+0000","last_peered":"2024-09-18T12:00:27.955646+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:27.955646+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27021929,"num_objects":6598,"num_object_clones":0,"num_object_copies":13196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12930,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3063},{"shards":"0,2","objects":369}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'12934","reported_seq":20082,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:28.708130+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:28.708130+0000","last_peered":"2024-09-18T12:00:28.708130+0000","last_clean":"2024-09-18T12:00:28.708130+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:28.708130+0000","last_undegraded":"2024-09-18T12:00:28.708130+0000","last_fullsized":"2024-09-18T12:00:28.708130+0000","mapping_epoch":40,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26986299,"num_objects":6591,"num_object_clones":0,"num_object_copies":13182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12931,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12827","reported_seq":12910,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.089899+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:28.089899+0000","last_peered":"2024-09-18T12:00:28.089899+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:28.089899+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26096850,"num_objects":6373,"num_object_clones":0,"num_object_copies":12746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12825,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2780},{"shards":"0,2","objects":367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'12880","reported_seq":12964,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:28.157128+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:28.157128+0000","last_peered":"2024-09-18T12:00:28.157128+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:28.157128+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'2800","ondisk_log_start":"30'2800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27167140,"num_objects":6636,"num_object_clones":0,"num_object_copies":13272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12877,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3113},{"shards":"0,2","objects":404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":429064268,"num_objects":104770,"num_object_clones":0,"num_object_copies":209540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33812,"num_objects_misplaced":352,"num_objects_unfound":0,"num_objects_dirty":104770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190445,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8428,"num_bytes_recovered":34411113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1017090048,"data_stored":1017019282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160952,"ondisk_log_size":160952,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051663,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633980,"kb_used_data":323620,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93737860,"statfs":{"total":96636764160,"available":95987568640,"internally_reserved":0,"allocated":331386880,"data_stored":331148764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.26200000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":1.7210000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855426,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597888,"kb_used_data":300020,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93773952,"statfs":{"total":96636764160,"available":96024526848,"internally_reserved":0,"allocated":307220480,"data_stored":306926557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":3.4860000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.7690000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.32300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674004,"kb_used_data":372036,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93697836,"statfs":{"total":96636764160,"available":95946584064,"internally_reserved":0,"allocated":380964864,"data_stored":380727198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,5,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.1500000000000004},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.32600000000000001}]},{"osd":2,"last update":"Wed Sep 18 11:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.1679999999999999,"15min":0.70199999999999996},"min":{"1min":0.23100000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":6.7720000000000002,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.217},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.0700000000000001,"15min":0.57099999999999995},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":9.7970000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":4.1890000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379662336,"data_stored":379658857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306720768,"data_stored":306661032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330706944,"data_stored":330699393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:33.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:00:33.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'12962", 'reported_seq': 13035, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:28.161018+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:00:28.161018+0000', 'last_peered': '2024-09-18T12:00:28.161018+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:00:28.161018+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'2900", 'ondisk_log_start': "30'2900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26927104, 'num_objects': 6574, 'num_object_clones': 0, 'num_object_copies': 13148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12962, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2975}, {'shards': '0,2', 'objects': 405}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:33.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:00:33.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'12939", 'reported_seq': 13125, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:28.390408+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:00:28.390408+0000', 'last_peered': '2024-09-18T12:00:28.390408+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:00:28.390408+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'2900", 'ondisk_log_start': "30'2900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27107352, 'num_objects': 6619, 'num_object_clones': 0, 'num_object_copies': 13238, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6619, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9797, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 377}, {'shards': '2', 'objects': 3083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:33.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:00:33.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'12871", 'reported_seq': 12954, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:28.113100+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:00:28.113100+0000', 'last_peered': '2024-09-18T12:00:28.113100+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:00:28.113100+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'2800", 'ondisk_log_start': "30'2800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26838226, 'num_objects': 6554, 'num_object_clones': 0, 'num_object_copies': 13108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12869, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3027}, {'shards': '0,2', 'objects': 370}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:33.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:00:33.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'12856", 'reported_seq': 12945, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:28.124592+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:00:28.124592+0000', 'last_peered': '2024-09-18T12:00:28.124592+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:00:28.124592+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'2800", 'ondisk_log_start': "30'2800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26690770, 'num_objects': 6518, 'num_object_clones': 0, 'num_object_copies': 13036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12854, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2959}, {'shards': '0,2', 'objects': 392}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:33.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:00:33.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'12867", 'reported_seq': 12947, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:28.111813+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:00:28.111813+0000', 'last_peered': '2024-09-18T12:00:28.111813+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:00:28.111813+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'2800", 'ondisk_log_start': "30'2800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26948818, 'num_objects': 6581, 'num_object_clones': 0, 'num_object_copies': 13162, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12865, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3049}, {'shards': '0,2', 'objects': 392}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:33.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:00:33.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'12940", 'reported_seq': 12998, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:28.272886+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:00:28.272886+0000', 'last_peered': '2024-09-18T12:00:28.272886+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:00:28.272886+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'2900", 'ondisk_log_start': "30'2900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26719442, 'num_objects': 6525, 'num_object_clones': 0, 'num_object_copies': 13050, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12938, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 344}, {'shards': '2', 'objects': 2975}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:33.971 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:00:33.971 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'12984", 'reported_seq': 15994, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:28.342593+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:00:28.342593+0000', 'last_peered': '2024-09-18T12:00:28.342593+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:00:28.342593+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'2900", 'ondisk_log_start': "30'2900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26681344, 'num_objects': 6514, 'num_object_clones': 0, 'num_object_copies': 13028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1843, 'num_objects_misplaced': 352, 'num_objects_unfound': 0, 'num_objects_dirty': 6514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9464, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1400, 'num_bytes_recovered': 5726208, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 352}, {'shards': '2', 'objects': 4319}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:33.971 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:00:33.971 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'12931", 'reported_seq': 13015, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:27.955646+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:00:27.955646+0000', 'last_peered': '2024-09-18T12:00:27.955646+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:00:27.955646+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'2900", 'ondisk_log_start': "30'2900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27021929, 'num_objects': 6598, 'num_object_clones': 0, 'num_object_copies': 13196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12930, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3063}, {'shards': '0,2', 'objects': 369}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:33.971 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:00:33.971 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'12827", 'reported_seq': 12910, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:28.089899+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:00:28.089899+0000', 'last_peered': '2024-09-18T12:00:28.089899+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:00:28.089899+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'2800", 'ondisk_log_start': "30'2800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26096850, 'num_objects': 6373, 'num_object_clones': 0, 'num_object_copies': 12746, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12825, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2780}, {'shards': '0,2', 'objects': 367}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:33.971 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:00:33.971 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'12880", 'reported_seq': 12964, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:28.157128+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:00:28.157128+0000', 'last_peered': '2024-09-18T12:00:28.157128+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:00:28.157128+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'2800", 'ondisk_log_start': "30'2800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27167140, 'num_objects': 6636, 'num_object_clones': 0, 'num_object_copies': 13272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12877, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3113}, {'shards': '0,2', 'objects': 404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:33.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:00:34.837 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:34.837 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33812/209544 objects degraded (16.136%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":434,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":104772,"data_bytes":429064287,"bytes_used":1951612928,"bytes_avail":287958679552,"bytes_total":289910292480,"degraded_objects":33812,"degraded_total":209544,"degraded_ratio":0.16135990531821476,"misplaced_objects":352,"misplaced_total":209544,"misplaced_ratio":0.0016798381246898024,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":79147,"recovering_keys_per_sec":0,"num_objects_recovered":232,"num_bytes_recovered":950272,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":354},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:00:34.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:00:34.870 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:35.634 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:35.637 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:35.657 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-09-18T12:00:35.176132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":420347999,"num_objects":102644,"num_object_clones":0,"num_object_copies":205288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33828,"num_objects_misplaced":217,"num_objects_unfound":0,"num_objects_dirty":102644,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192575,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8548,"num_bytes_recovered":34898556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160783,"ondisk_log_size":160783,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1910840,"kb_used_data":984028,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281204680,"statfs":{"total":289910292480,"available":287953592320,"internally_reserved":0,"allocated":1007644672,"data_stored":1006874967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,2,5,8,12,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21098496,"num_objects":-5151,"num_object_clones":0,"num_object_copies":-10302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39,"num_objects_misplaced":-390,"num_objects_unfound":0,"num_objects_dirty":-5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5151,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1437696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002749"},"pg_stats":[{"pgid":"2.f","version":"41'13098","reported_seq":13171,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:33.214286+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:33.214286+0000","last_peered":"2024-09-18T12:00:33.214286+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:33.214286+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26370048,"num_objects":6438,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13098,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2839},{"shards":"0,2","objects":269}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13037","reported_seq":13197,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:33.382350+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:33.382350+0000","last_peered":"2024-09-18T12:00:33.382350+0000","last_clean":"2024-09-18T12:00:33.382350+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:33.382350+0000","last_undegraded":"2024-09-18T12:00:33.382350+0000","last_fullsized":"2024-09-18T12:00:33.382350+0000","mapping_epoch":29,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26193920,"num_objects":6395,"num_object_clones":0,"num_object_copies":12790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10229,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13049","reported_seq":13237,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:33.345790+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:33.345790+0000","last_peered":"2024-09-18T12:00:33.345790+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:33.345790+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26656792,"num_objects":6509,"num_object_clones":0,"num_object_copies":13018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9907,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":267},{"shards":"2","objects":2973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'12910","reported_seq":12941,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:33.177432+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:33.177432+0000","last_peered":"2024-09-18T12:00:33.177432+0000","last_clean":"2024-09-18T12:00:33.177432+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:33.177432+0000","last_undegraded":"2024-09-18T12:00:33.177432+0000","last_fullsized":"2024-09-18T12:00:33.177432+0000","mapping_epoch":29,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26009600,"num_objects":6350,"num_object_clones":0,"num_object_copies":12700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12910,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13010","reported_seq":13203,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:33.166474+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:33.166474+0000","last_peered":"2024-09-18T12:00:33.166474+0000","last_clean":"2024-09-18T12:00:33.166474+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:33.166474+0000","last_undegraded":"2024-09-18T12:00:33.166474+0000","last_fullsized":"2024-09-18T12:00:33.166474+0000","mapping_epoch":29,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26365952,"num_objects":6437,"num_object_clones":0,"num_object_copies":12874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10175,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'12997","reported_seq":13080,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:33.164688+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:33.164688+0000","last_peered":"2024-09-18T12:00:33.164688+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:33.164688+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26322130,"num_objects":6428,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12995,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2901},{"shards":"0,2","objects":244}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'12992","reported_seq":13081,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:33.174193+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:33.174193+0000","last_peered":"2024-09-18T12:00:33.174193+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:33.174193+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26133714,"num_objects":6382,"num_object_clones":0,"num_object_copies":12764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12990,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2823},{"shards":"0,2","objects":256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13222","reported_seq":13449,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:33.149086+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:33.149086+0000","last_peered":"2024-09-18T12:00:33.149086+0000","last_clean":"2024-09-18T12:00:33.149086+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:33.149086+0000","last_undegraded":"2024-09-18T12:00:33.149086+0000","last_fullsized":"2024-09-18T12:00:33.149086+0000","mapping_epoch":29,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26132480,"num_objects":6380,"num_object_clones":0,"num_object_copies":12760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9539,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'12943","reported_seq":20073,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:33.924596+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:33.924596+0000","last_peered":"2024-09-18T12:00:33.924596+0000","last_clean":"2024-09-18T12:00:33.924596+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:33.924596+0000","last_undegraded":"2024-09-18T12:00:33.924596+0000","last_fullsized":"2024-09-18T12:00:33.924596+0000","mapping_epoch":38,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26469586,"num_objects":6464,"num_object_clones":0,"num_object_copies":12928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12941,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13010","reported_seq":13090,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:33.163086+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:33.163086+0000","last_peered":"2024-09-18T12:00:33.163086+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:33.163086+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26363090,"num_objects":6438,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13008,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2906},{"shards":"0,2","objects":249}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13057","reported_seq":13115,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:33.284629+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:33.284629+0000","last_peered":"2024-09-18T12:00:33.284629+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:33.284629+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26240210,"num_objects":6408,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13055,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":227},{"shards":"2","objects":2858}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13126","reported_seq":16375,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:33.409194+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:33.409194+0000","last_peered":"2024-09-18T12:00:33.409194+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:33.409194+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26099712,"num_objects":6372,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":217,"num_objects_unfound":0,"num_objects_dirty":6372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9606,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1519,"num_bytes_recovered":6213632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":217},{"shards":"2","objects":4296}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13074","reported_seq":13158,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:33.175517+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:33.175517+0000","last_peered":"2024-09-18T12:00:33.175517+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:33.175517+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26436201,"num_objects":6455,"num_object_clones":0,"num_object_copies":12910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13073,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2920},{"shards":"0,2","objects":226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13070","reported_seq":20218,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:33.902247+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:33.902247+0000","last_peered":"2024-09-18T12:00:33.902247+0000","last_clean":"2024-09-18T12:00:33.902247+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:33.902247+0000","last_undegraded":"2024-09-18T12:00:33.902247+0000","last_fullsized":"2024-09-18T12:00:33.902247+0000","mapping_epoch":40,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26429243,"num_objects":6455,"num_object_clones":0,"num_object_copies":12910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13067,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'12968","reported_seq":13046,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:33.187218+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:33.187218+0000","last_peered":"2024-09-18T12:00:33.187218+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:33.187218+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'2900","ondisk_log_start":"30'2900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25519314,"num_objects":6232,"num_object_clones":0,"num_object_copies":12464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12966,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2639},{"shards":"0,2","objects":226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13017","reported_seq":13104,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:33.142539+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:33.142539+0000","last_peered":"2024-09-18T12:00:33.142539+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:33.142539+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26605988,"num_objects":6499,"num_object_clones":0,"num_object_copies":12998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13014,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2976},{"shards":"0,2","objects":267}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":420347980,"num_objects":102642,"num_object_clones":0,"num_object_copies":205284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33828,"num_objects_misplaced":217,"num_objects_unfound":0,"num_objects_dirty":102642,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192573,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8547,"num_bytes_recovered":34898537,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1005002752,"data_stored":1004931986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160780,"ondisk_log_size":160780,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051664,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633900,"kb_used_data":319392,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93737940,"statfs":{"total":96636764160,"available":95987650560,"internally_reserved":0,"allocated":327057408,"data_stored":326819292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.50800000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.29999999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855427,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601004,"kb_used_data":298980,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93770836,"statfs":{"total":96636764160,"available":96021336064,"internally_reserved":0,"allocated":306155520,"data_stored":305861597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":3.4860000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.7690000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.32300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675936,"kb_used_data":365656,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93695904,"statfs":{"total":96636764160,"available":95944605696,"internally_reserved":0,"allocated":374431744,"data_stored":374194078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,2,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.30499999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":372908032,"data_stored":372904553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305668096,"data_stored":305608360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":326426624,"data_stored":326419073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:38.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:39.440 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:39.445 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:39.462 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-09-18T12:00:39.177040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":411639903,"num_objects":100518,"num_object_clones":0,"num_object_copies":201036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33828,"num_objects_misplaced":81,"num_objects_unfound":0,"num_objects_dirty":100518,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194701,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8684,"num_bytes_recovered":35455612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160909,"ondisk_log_size":160909,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1882704,"kb_used_data":972532,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281232816,"statfs":{"total":289910292480,"available":287982403584,"internally_reserved":0,"allocated":995872768,"data_stored":995103063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,5,5,6,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26210304,"num_objects":-6399,"num_object_clones":0,"num_object_copies":-12798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":-405,"num_objects_unfound":0,"num_objects_dirty":-6399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6399,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":370,"num_bytes_recovered":1515520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002702"},"pg_stats":[{"pgid":"2.f","version":"41'13252","reported_seq":13324,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.163965+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:38.163965+0000","last_peered":"2024-09-18T12:00:38.163965+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:38.163965+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25739264,"num_objects":6284,"num_object_clones":0,"num_object_copies":12568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13252,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2685},{"shards":"0,2","objects":115}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13160","reported_seq":13320,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.219957+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:38.219957+0000","last_peered":"2024-09-18T12:00:38.219957+0000","last_clean":"2024-09-18T12:00:38.219957+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:38.219957+0000","last_undegraded":"2024-09-18T12:00:38.219957+0000","last_fullsized":"2024-09-18T12:00:38.219957+0000","mapping_epoch":29,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":6272,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10352,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13206","reported_seq":13392,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.349509+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:38.349509+0000","last_peered":"2024-09-18T12:00:38.349509+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:38.349509+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26013720,"num_objects":6352,"num_object_clones":0,"num_object_copies":12704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10064,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":110},{"shards":"2","objects":2816}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13033","reported_seq":13064,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.358757+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:38.358757+0000","last_peered":"2024-09-18T12:00:38.358757+0000","last_clean":"2024-09-18T12:00:38.358757+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:38.358757+0000","last_undegraded":"2024-09-18T12:00:38.358757+0000","last_fullsized":"2024-09-18T12:00:38.358757+0000","mapping_epoch":29,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25505792,"num_objects":6227,"num_object_clones":0,"num_object_copies":12454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13033,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13150","reported_seq":13341,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.194251+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:38.194251+0000","last_peered":"2024-09-18T12:00:38.194251+0000","last_clean":"2024-09-18T12:00:38.194251+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:38.194251+0000","last_undegraded":"2024-09-18T12:00:38.194251+0000","last_fullsized":"2024-09-18T12:00:38.194251+0000","mapping_epoch":29,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25792512,"num_objects":6297,"num_object_clones":0,"num_object_copies":12594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10315,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13125","reported_seq":13208,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.137305+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:38.137305+0000","last_peered":"2024-09-18T12:00:38.137305+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:38.137305+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25797842,"num_objects":6300,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13123,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2773},{"shards":"0,2","objects":116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13134","reported_seq":13223,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:37.953236+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:37.953236+0000","last_peered":"2024-09-18T12:00:37.953236+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:37.953236+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25552082,"num_objects":6240,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13132,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2681},{"shards":"0,2","objects":114}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13361","reported_seq":13588,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.181354+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:38.181354+0000","last_peered":"2024-09-18T12:00:38.181354+0000","last_clean":"2024-09-18T12:00:38.181354+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:38.181354+0000","last_undegraded":"2024-09-18T12:00:38.181354+0000","last_fullsized":"2024-09-18T12:00:38.181354+0000","mapping_epoch":29,"log_start":"30'3300","ondisk_log_start":"30'3300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25563136,"num_objects":6241,"num_object_clones":0,"num_object_copies":12482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13090","reported_seq":20220,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.998852+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:38.998852+0000","last_peered":"2024-09-18T12:00:38.998852+0000","last_clean":"2024-09-18T12:00:38.998852+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:38.998852+0000","last_undegraded":"2024-09-18T12:00:38.998852+0000","last_fullsized":"2024-09-18T12:00:38.998852+0000","mapping_epoch":38,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25867474,"num_objects":6317,"num_object_clones":0,"num_object_copies":12634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13088,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13154","reported_seq":13234,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.162161+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:38.162161+0000","last_peered":"2024-09-18T12:00:38.162161+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:38.162161+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25773266,"num_objects":6294,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13152,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2762},{"shards":"0,2","objects":105}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13178","reported_seq":13236,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.309783+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:38.309783+0000","last_peered":"2024-09-18T12:00:38.309783+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:38.309783+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744594,"num_objects":6287,"num_object_clones":0,"num_object_copies":12574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13176,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":106},{"shards":"2","objects":2737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13241","reported_seq":16763,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.403207+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:38.403207+0000","last_peered":"2024-09-18T12:00:38.403207+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:38.403207+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25628672,"num_objects":6257,"num_object_clones":0,"num_object_copies":12514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":81,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9721,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":6770688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":81},{"shards":"2","objects":4317}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13187","reported_seq":13271,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.186925+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:38.186925+0000","last_peered":"2024-09-18T12:00:38.186925+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:38.186925+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25973353,"num_objects":6342,"num_object_clones":0,"num_object_copies":12684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13186,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2807},{"shards":"0,2","objects":113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13199","reported_seq":20347,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.840107+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:38.840107+0000","last_peered":"2024-09-18T12:00:38.840107+0000","last_clean":"2024-09-18T12:00:38.840107+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:38.840107+0000","last_undegraded":"2024-09-18T12:00:38.840107+0000","last_fullsized":"2024-09-18T12:00:38.840107+0000","mapping_epoch":40,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25900859,"num_objects":6326,"num_object_clones":0,"num_object_copies":12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13196,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13080","reported_seq":13161,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.135070+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:38.135070+0000","last_peered":"2024-09-18T12:00:38.135070+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:38.135070+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25060562,"num_objects":6120,"num_object_clones":0,"num_object_copies":12240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13078,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2527},{"shards":"0,2","objects":114}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13156","reported_seq":13243,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.195652+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:38.195652+0000","last_peered":"2024-09-18T12:00:38.195652+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:38.195652+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26036644,"num_objects":6360,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2837},{"shards":"0,2","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":411639884,"num_objects":100516,"num_object_clones":0,"num_object_copies":201032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33828,"num_objects_misplaced":81,"num_objects_unfound":0,"num_objects_dirty":100516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194699,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8683,"num_bytes_recovered":35455593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":993165312,"data_stored":993094546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051665,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617260,"kb_used_data":315236,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93754580,"statfs":{"total":96636764160,"available":96004689920,"internally_reserved":0,"allocated":322801664,"data_stored":322563548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.628}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.29199999999999998},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.47699999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855428,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608288,"kb_used_data":297940,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93763552,"statfs":{"total":96636764160,"available":96013877248,"internally_reserved":0,"allocated":305090560,"data_stored":304796637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.191},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.22600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":657156,"kb_used_data":359356,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93714684,"statfs":{"total":96636764160,"available":95963836416,"internally_reserved":0,"allocated":367980544,"data_stored":367742878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":1.399},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":11.957000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":366424064,"data_stored":366420585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":304652288,"data_stored":304592552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322088960,"data_stored":322081409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:39.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:40.215 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:40.230 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:40.244 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-09-18T12:00:39.177040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":411639903,"num_objects":100518,"num_object_clones":0,"num_object_copies":201036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33828,"num_objects_misplaced":81,"num_objects_unfound":0,"num_objects_dirty":100518,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194701,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8684,"num_bytes_recovered":35455612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160909,"ondisk_log_size":160909,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1882704,"kb_used_data":972532,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281232816,"statfs":{"total":289910292480,"available":287982403584,"internally_reserved":0,"allocated":995872768,"data_stored":995103063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,5,5,6,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26210304,"num_objects":-6399,"num_object_clones":0,"num_object_copies":-12798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":-405,"num_objects_unfound":0,"num_objects_dirty":-6399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6399,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":370,"num_bytes_recovered":1515520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002702"},"pg_stats":[{"pgid":"2.f","version":"41'13252","reported_seq":13324,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.163965+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:38.163965+0000","last_peered":"2024-09-18T12:00:38.163965+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:38.163965+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25739264,"num_objects":6284,"num_object_clones":0,"num_object_copies":12568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13252,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2685},{"shards":"0,2","objects":115}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13160","reported_seq":13320,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.219957+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:38.219957+0000","last_peered":"2024-09-18T12:00:38.219957+0000","last_clean":"2024-09-18T12:00:38.219957+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:38.219957+0000","last_undegraded":"2024-09-18T12:00:38.219957+0000","last_fullsized":"2024-09-18T12:00:38.219957+0000","mapping_epoch":29,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":6272,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10352,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13206","reported_seq":13392,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.349509+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:38.349509+0000","last_peered":"2024-09-18T12:00:38.349509+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:38.349509+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26013720,"num_objects":6352,"num_object_clones":0,"num_object_copies":12704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10064,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":110},{"shards":"2","objects":2816}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13033","reported_seq":13064,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.358757+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:38.358757+0000","last_peered":"2024-09-18T12:00:38.358757+0000","last_clean":"2024-09-18T12:00:38.358757+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:38.358757+0000","last_undegraded":"2024-09-18T12:00:38.358757+0000","last_fullsized":"2024-09-18T12:00:38.358757+0000","mapping_epoch":29,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25505792,"num_objects":6227,"num_object_clones":0,"num_object_copies":12454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13033,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13150","reported_seq":13341,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.194251+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:38.194251+0000","last_peered":"2024-09-18T12:00:38.194251+0000","last_clean":"2024-09-18T12:00:38.194251+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:38.194251+0000","last_undegraded":"2024-09-18T12:00:38.194251+0000","last_fullsized":"2024-09-18T12:00:38.194251+0000","mapping_epoch":29,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25792512,"num_objects":6297,"num_object_clones":0,"num_object_copies":12594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10315,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13125","reported_seq":13208,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.137305+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:38.137305+0000","last_peered":"2024-09-18T12:00:38.137305+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:38.137305+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25797842,"num_objects":6300,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13123,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2773},{"shards":"0,2","objects":116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13134","reported_seq":13223,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:37.953236+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:37.953236+0000","last_peered":"2024-09-18T12:00:37.953236+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:37.953236+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25552082,"num_objects":6240,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13132,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2681},{"shards":"0,2","objects":114}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13361","reported_seq":13588,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.181354+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:38.181354+0000","last_peered":"2024-09-18T12:00:38.181354+0000","last_clean":"2024-09-18T12:00:38.181354+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:38.181354+0000","last_undegraded":"2024-09-18T12:00:38.181354+0000","last_fullsized":"2024-09-18T12:00:38.181354+0000","mapping_epoch":29,"log_start":"30'3300","ondisk_log_start":"30'3300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25563136,"num_objects":6241,"num_object_clones":0,"num_object_copies":12482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13090","reported_seq":20220,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.998852+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:38.998852+0000","last_peered":"2024-09-18T12:00:38.998852+0000","last_clean":"2024-09-18T12:00:38.998852+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:38.998852+0000","last_undegraded":"2024-09-18T12:00:38.998852+0000","last_fullsized":"2024-09-18T12:00:38.998852+0000","mapping_epoch":38,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25867474,"num_objects":6317,"num_object_clones":0,"num_object_copies":12634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13088,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13154","reported_seq":13234,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.162161+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:38.162161+0000","last_peered":"2024-09-18T12:00:38.162161+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:38.162161+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25773266,"num_objects":6294,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13152,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2762},{"shards":"0,2","objects":105}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13178","reported_seq":13236,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.309783+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:38.309783+0000","last_peered":"2024-09-18T12:00:38.309783+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:38.309783+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744594,"num_objects":6287,"num_object_clones":0,"num_object_copies":12574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13176,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":106},{"shards":"2","objects":2737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13241","reported_seq":16763,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.403207+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:38.403207+0000","last_peered":"2024-09-18T12:00:38.403207+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:38.403207+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25628672,"num_objects":6257,"num_object_clones":0,"num_object_copies":12514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":81,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9721,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":6770688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":81},{"shards":"2","objects":4317}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13187","reported_seq":13271,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.186925+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:38.186925+0000","last_peered":"2024-09-18T12:00:38.186925+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:38.186925+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25973353,"num_objects":6342,"num_object_clones":0,"num_object_copies":12684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13186,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2807},{"shards":"0,2","objects":113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13199","reported_seq":20347,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.840107+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:38.840107+0000","last_peered":"2024-09-18T12:00:38.840107+0000","last_clean":"2024-09-18T12:00:38.840107+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:38.840107+0000","last_undegraded":"2024-09-18T12:00:38.840107+0000","last_fullsized":"2024-09-18T12:00:38.840107+0000","mapping_epoch":40,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25900859,"num_objects":6326,"num_object_clones":0,"num_object_copies":12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13196,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13080","reported_seq":13161,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.135070+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:38.135070+0000","last_peered":"2024-09-18T12:00:38.135070+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:38.135070+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25060562,"num_objects":6120,"num_object_clones":0,"num_object_copies":12240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13078,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2527},{"shards":"0,2","objects":114}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13156","reported_seq":13243,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.195652+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:38.195652+0000","last_peered":"2024-09-18T12:00:38.195652+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:38.195652+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26036644,"num_objects":6360,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2837},{"shards":"0,2","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":411639884,"num_objects":100516,"num_object_clones":0,"num_object_copies":201032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33828,"num_objects_misplaced":81,"num_objects_unfound":0,"num_objects_dirty":100516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194699,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8683,"num_bytes_recovered":35455593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":993165312,"data_stored":993094546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051665,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617260,"kb_used_data":315236,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93754580,"statfs":{"total":96636764160,"available":96004689920,"internally_reserved":0,"allocated":322801664,"data_stored":322563548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.628}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.29199999999999998},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.47699999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855428,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608288,"kb_used_data":297940,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93763552,"statfs":{"total":96636764160,"available":96013877248,"internally_reserved":0,"allocated":305090560,"data_stored":304796637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.191},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.22600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":657156,"kb_used_data":359356,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93714684,"statfs":{"total":96636764160,"available":95963836416,"internally_reserved":0,"allocated":367980544,"data_stored":367742878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":1.399},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":11.957000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":366424064,"data_stored":366420585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":304652288,"data_stored":304592552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322088960,"data_stored":322081409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:40.249 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:00:40.249 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'13252", 'reported_seq': 13324, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:38.163965+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:00:38.163965+0000', 'last_peered': '2024-09-18T12:00:38.163965+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:00:38.163965+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "30'3200", 'ondisk_log_start': "30'3200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25739264, 'num_objects': 6284, 'num_object_clones': 0, 'num_object_copies': 12568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13252, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2685}, {'shards': '0,2', 'objects': 115}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:40.249 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:00:40.249 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'13206", 'reported_seq': 13392, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:38.349509+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:00:38.349509+0000', 'last_peered': '2024-09-18T12:00:38.349509+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:00:38.349509+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "30'3200", 'ondisk_log_start': "30'3200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26013720, 'num_objects': 6352, 'num_object_clones': 0, 'num_object_copies': 12704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6352, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10064, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 110}, {'shards': '2', 'objects': 2816}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:40.249 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:00:40.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'13125", 'reported_seq': 13208, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:38.137305+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:00:38.137305+0000', 'last_peered': '2024-09-18T12:00:38.137305+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:00:38.137305+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'3100", 'ondisk_log_start': "30'3100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25797842, 'num_objects': 6300, 'num_object_clones': 0, 'num_object_copies': 12600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13123, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2773}, {'shards': '0,2', 'objects': 116}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:40.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:00:40.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'13134", 'reported_seq': 13223, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:37.953236+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:00:37.953236+0000', 'last_peered': '2024-09-18T12:00:37.953236+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:00:37.953236+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'3100", 'ondisk_log_start': "30'3100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25552082, 'num_objects': 6240, 'num_object_clones': 0, 'num_object_copies': 12480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13132, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2681}, {'shards': '0,2', 'objects': 114}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:40.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:00:40.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'13154", 'reported_seq': 13234, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:38.162161+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:00:38.162161+0000', 'last_peered': '2024-09-18T12:00:38.162161+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:00:38.162161+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'3100", 'ondisk_log_start': "30'3100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25773266, 'num_objects': 6294, 'num_object_clones': 0, 'num_object_copies': 12588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13152, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2762}, {'shards': '0,2', 'objects': 105}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:40.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:00:40.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'13178", 'reported_seq': 13236, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:38.309783+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:00:38.309783+0000', 'last_peered': '2024-09-18T12:00:38.309783+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:00:38.309783+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "30'3100", 'ondisk_log_start': "30'3100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25744594, 'num_objects': 6287, 'num_object_clones': 0, 'num_object_copies': 12574, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13176, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 106}, {'shards': '2', 'objects': 2737}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:40.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:00:40.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'13241", 'reported_seq': 16763, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:38.403207+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:00:38.403207+0000', 'last_peered': '2024-09-18T12:00:38.403207+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:00:38.403207+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'3200", 'ondisk_log_start': "30'3200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25628672, 'num_objects': 6257, 'num_object_clones': 0, 'num_object_copies': 12514, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1859, 'num_objects_misplaced': 81, 'num_objects_unfound': 0, 'num_objects_dirty': 6257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9721, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1655, 'num_bytes_recovered': 6770688, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 81}, {'shards': '2', 'objects': 4317}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:40.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:00:40.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'13187", 'reported_seq': 13271, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:38.186925+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:00:38.186925+0000', 'last_peered': '2024-09-18T12:00:38.186925+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:00:38.186925+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "30'3100", 'ondisk_log_start': "30'3100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25973353, 'num_objects': 6342, 'num_object_clones': 0, 'num_object_copies': 12684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13186, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2807}, {'shards': '0,2', 'objects': 113}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:40.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:00:40.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'13080", 'reported_seq': 13161, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:38.135070+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:00:38.135070+0000', 'last_peered': '2024-09-18T12:00:38.135070+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:00:38.135070+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'3000", 'ondisk_log_start': "30'3000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25060562, 'num_objects': 6120, 'num_object_clones': 0, 'num_object_copies': 12240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13078, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2527}, {'shards': '0,2', 'objects': 114}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:40.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:00:40.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'13156", 'reported_seq': 13243, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:38.195652+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:00:38.195652+0000', 'last_peered': '2024-09-18T12:00:38.195652+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:00:38.195652+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "30'3100", 'ondisk_log_start': "30'3100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26036644, 'num_objects': 6360, 'num_object_clones': 0, 'num_object_copies': 12720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13153, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2837}, {'shards': '0,2', 'objects': 128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:40.251 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:00:41.127 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:41.127 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33828/201036 objects degraded (16.827%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":441,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":100518,"data_bytes":411639903,"bytes_used":1927888896,"bytes_avail":287982403584,"bytes_total":289910292480,"degraded_objects":33828,"degraded_total":201036,"degraded_ratio":0.16826836984420701,"misplaced_objects":81,"misplaced_total":201036,"misplaced_ratio":0.00040291291112039637,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":126264,"recovering_keys_per_sec":0,"num_objects_recovered":370,"num_bytes_recovered":1515520,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":533},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:00:41.160 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:00:41.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:42.017 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:42.021 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:42.043 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-18T12:00:41.177530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":411639903,"num_objects":100518,"num_object_clones":0,"num_object_copies":201036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33828,"num_objects_misplaced":81,"num_objects_unfound":0,"num_objects_dirty":100518,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194701,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8684,"num_bytes_recovered":35455612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160909,"ondisk_log_size":160909,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1882704,"kb_used_data":972532,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281232816,"statfs":{"total":289910292480,"available":287982403584,"internally_reserved":0,"allocated":995872768,"data_stored":995103063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,5,5,6,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17424384,"num_objects":-4254,"num_object_clones":0,"num_object_copies":-8508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":-271,"num_objects_unfound":0,"num_objects_dirty":-4254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4254,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":1044480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002720"},"pg_stats":[{"pgid":"2.f","version":"41'13252","reported_seq":13324,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.163965+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:38.163965+0000","last_peered":"2024-09-18T12:00:38.163965+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:38.163965+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25739264,"num_objects":6284,"num_object_clones":0,"num_object_copies":12568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13252,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2685},{"shards":"0,2","objects":115}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13160","reported_seq":13320,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.219957+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:38.219957+0000","last_peered":"2024-09-18T12:00:38.219957+0000","last_clean":"2024-09-18T12:00:38.219957+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:38.219957+0000","last_undegraded":"2024-09-18T12:00:38.219957+0000","last_fullsized":"2024-09-18T12:00:38.219957+0000","mapping_epoch":29,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":6272,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10352,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13206","reported_seq":13392,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.349509+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:38.349509+0000","last_peered":"2024-09-18T12:00:38.349509+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:38.349509+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26013720,"num_objects":6352,"num_object_clones":0,"num_object_copies":12704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10064,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":110},{"shards":"2","objects":2816}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13033","reported_seq":13064,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.358757+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:38.358757+0000","last_peered":"2024-09-18T12:00:38.358757+0000","last_clean":"2024-09-18T12:00:38.358757+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:38.358757+0000","last_undegraded":"2024-09-18T12:00:38.358757+0000","last_fullsized":"2024-09-18T12:00:38.358757+0000","mapping_epoch":29,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25505792,"num_objects":6227,"num_object_clones":0,"num_object_copies":12454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13033,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13150","reported_seq":13341,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.194251+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:38.194251+0000","last_peered":"2024-09-18T12:00:38.194251+0000","last_clean":"2024-09-18T12:00:38.194251+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:38.194251+0000","last_undegraded":"2024-09-18T12:00:38.194251+0000","last_fullsized":"2024-09-18T12:00:38.194251+0000","mapping_epoch":29,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25792512,"num_objects":6297,"num_object_clones":0,"num_object_copies":12594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10315,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13125","reported_seq":13208,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.137305+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:38.137305+0000","last_peered":"2024-09-18T12:00:38.137305+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:38.137305+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25797842,"num_objects":6300,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13123,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2773},{"shards":"0,2","objects":116}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13134","reported_seq":13223,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:37.953236+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:37.953236+0000","last_peered":"2024-09-18T12:00:37.953236+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:37.953236+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25552082,"num_objects":6240,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13132,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2681},{"shards":"0,2","objects":114}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13361","reported_seq":13588,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.181354+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:38.181354+0000","last_peered":"2024-09-18T12:00:38.181354+0000","last_clean":"2024-09-18T12:00:38.181354+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:38.181354+0000","last_undegraded":"2024-09-18T12:00:38.181354+0000","last_fullsized":"2024-09-18T12:00:38.181354+0000","mapping_epoch":29,"log_start":"30'3300","ondisk_log_start":"30'3300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25563136,"num_objects":6241,"num_object_clones":0,"num_object_copies":12482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13090","reported_seq":20220,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.998852+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:38.998852+0000","last_peered":"2024-09-18T12:00:38.998852+0000","last_clean":"2024-09-18T12:00:38.998852+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:38.998852+0000","last_undegraded":"2024-09-18T12:00:38.998852+0000","last_fullsized":"2024-09-18T12:00:38.998852+0000","mapping_epoch":38,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25867474,"num_objects":6317,"num_object_clones":0,"num_object_copies":12634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13088,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13154","reported_seq":13234,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.162161+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:38.162161+0000","last_peered":"2024-09-18T12:00:38.162161+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:38.162161+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25773266,"num_objects":6294,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13152,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2762},{"shards":"0,2","objects":105}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13178","reported_seq":13236,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.309783+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:38.309783+0000","last_peered":"2024-09-18T12:00:38.309783+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:38.309783+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25744594,"num_objects":6287,"num_object_clones":0,"num_object_copies":12574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13176,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":106},{"shards":"2","objects":2737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13241","reported_seq":16763,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.403207+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:38.403207+0000","last_peered":"2024-09-18T12:00:38.403207+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:38.403207+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25628672,"num_objects":6257,"num_object_clones":0,"num_object_copies":12514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1859,"num_objects_misplaced":81,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9721,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":6770688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":81},{"shards":"2","objects":4317}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13187","reported_seq":13271,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.186925+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:38.186925+0000","last_peered":"2024-09-18T12:00:38.186925+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:38.186925+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25973353,"num_objects":6342,"num_object_clones":0,"num_object_copies":12684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13186,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2807},{"shards":"0,2","objects":113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13199","reported_seq":20347,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:38.840107+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:38.840107+0000","last_peered":"2024-09-18T12:00:38.840107+0000","last_clean":"2024-09-18T12:00:38.840107+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:38.840107+0000","last_undegraded":"2024-09-18T12:00:38.840107+0000","last_fullsized":"2024-09-18T12:00:38.840107+0000","mapping_epoch":40,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25900859,"num_objects":6326,"num_object_clones":0,"num_object_copies":12652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13196,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13080","reported_seq":13161,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.135070+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:38.135070+0000","last_peered":"2024-09-18T12:00:38.135070+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:38.135070+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'3000","ondisk_log_start":"30'3000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25060562,"num_objects":6120,"num_object_clones":0,"num_object_copies":12240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13078,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2527},{"shards":"0,2","objects":114}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13156","reported_seq":13243,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:38.195652+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:38.195652+0000","last_peered":"2024-09-18T12:00:38.195652+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:38.195652+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26036644,"num_objects":6360,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2837},{"shards":"0,2","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":411639884,"num_objects":100516,"num_object_clones":0,"num_object_copies":201032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33828,"num_objects_misplaced":81,"num_objects_unfound":0,"num_objects_dirty":100516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194699,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8683,"num_bytes_recovered":35455593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":993165312,"data_stored":993094546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051665,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617260,"kb_used_data":315236,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93754580,"statfs":{"total":96636764160,"available":96004689920,"internally_reserved":0,"allocated":322801664,"data_stored":322563548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.628}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.29199999999999998},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.47699999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855428,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608288,"kb_used_data":297940,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93763552,"statfs":{"total":96636764160,"available":96013877248,"internally_reserved":0,"allocated":305090560,"data_stored":304796637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.191},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.22600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":657156,"kb_used_data":359356,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93714684,"statfs":{"total":96636764160,"available":95963836416,"internally_reserved":0,"allocated":367980544,"data_stored":367742878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20499999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":1.399},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":11.957000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":366424064,"data_stored":366420585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":304652288,"data_stored":304592552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":322088960,"data_stored":322081409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:45.046 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:45.968 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:45.981 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-18T12:00:45.178349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":402858079,"num_objects":98374,"num_object_clones":0,"num_object_copies":196748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196845,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8824,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1861008,"kb_used_data":959156,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281254512,"statfs":{"total":289910292480,"available":288004620288,"internally_reserved":0,"allocated":982175744,"data_stored":981406039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,10,4,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26206208,"num_objects":-6398,"num_object_clones":0,"num_object_copies":-12796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-43,"num_objects_misplaced":-352,"num_objects_unfound":0,"num_objects_dirty":-6398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6398,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":395,"num_bytes_recovered":1380352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"2.f","version":"41'13384","reported_seq":13456,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.254169+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:43.254169+0000","last_peered":"2024-09-18T12:00:43.254169+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:43.254169+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25198592,"num_objects":6152,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13384,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2553}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13301","reported_seq":13459,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:43.339220+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:43.339220+0000","last_peered":"2024-09-18T12:00:43.339220+0000","last_clean":"2024-09-18T12:00:43.339220+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:43.339220+0000","last_undegraded":"2024-09-18T12:00:43.339220+0000","last_fullsized":"2024-09-18T12:00:43.339220+0000","mapping_epoch":29,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25112576,"num_objects":6131,"num_object_clones":0,"num_object_copies":12262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13339","reported_seq":13527,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.313970+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:43.313970+0000","last_peered":"2024-09-18T12:00:43.313970+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:43.313970+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25468952,"num_objects":6219,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10197,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2683}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13153","reported_seq":13183,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:43.376522+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:43.376522+0000","last_peered":"2024-09-18T12:00:43.376522+0000","last_clean":"2024-09-18T12:00:43.376522+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:43.376522+0000","last_undegraded":"2024-09-18T12:00:43.376522+0000","last_fullsized":"2024-09-18T12:00:43.376522+0000","mapping_epoch":29,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25014272,"num_objects":6107,"num_object_clones":0,"num_object_copies":12214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13279","reported_seq":13472,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:43.200818+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:43.200818+0000","last_peered":"2024-09-18T12:00:43.200818+0000","last_clean":"2024-09-18T12:00:43.200818+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:43.200818+0000","last_undegraded":"2024-09-18T12:00:43.200818+0000","last_fullsized":"2024-09-18T12:00:43.200818+0000","mapping_epoch":29,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25264128,"num_objects":6168,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10444,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13257","reported_seq":13340,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.195014+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:43.195014+0000","last_peered":"2024-09-18T12:00:43.195014+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:43.195014+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25257170,"num_objects":6168,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13255,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13267","reported_seq":13356,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.248357+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:43.248357+0000","last_peered":"2024-09-18T12:00:43.248357+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:43.248357+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25007314,"num_objects":6107,"num_object_clones":0,"num_object_copies":12214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13265,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13499","reported_seq":13726,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:43.143577+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:43.143577+0000","last_peered":"2024-09-18T12:00:43.143577+0000","last_clean":"2024-09-18T12:00:43.143577+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:43.143577+0000","last_undegraded":"2024-09-18T12:00:43.143577+0000","last_fullsized":"2024-09-18T12:00:43.143577+0000","mapping_epoch":29,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24997888,"num_objects":6103,"num_object_clones":0,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13240","reported_seq":20369,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:44.074421+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:44.074421+0000","last_peered":"2024-09-18T12:00:44.074421+0000","last_clean":"2024-09-18T12:00:44.074421+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:44.074421+0000","last_undegraded":"2024-09-18T12:00:44.074421+0000","last_fullsized":"2024-09-18T12:00:44.074421+0000","mapping_epoch":38,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25253074,"num_objects":6167,"num_object_clones":0,"num_object_copies":12334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13238,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13276","reported_seq":13356,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.179868+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:43.179868+0000","last_peered":"2024-09-18T12:00:43.179868+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:43.179868+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25273554,"num_objects":6172,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13274,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2640}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13311","reported_seq":13369,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.239231+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:43.239231+0000","last_peered":"2024-09-18T12:00:43.239231+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:43.239231+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25199826,"num_objects":6154,"num_object_clones":0,"num_object_copies":12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13309,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2604}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13362","reported_seq":17164,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.360208+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:43.360208+0000","last_peered":"2024-09-18T12:00:43.360208+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:43.360208+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'3300","ondisk_log_start":"30'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25133056,"num_objects":6136,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9842,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1795,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4336}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13323","reported_seq":13405,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.176765+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:43.176765+0000","last_peered":"2024-09-18T12:00:43.176765+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:43.176765+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25416297,"num_objects":6206,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13322,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13346","reported_seq":20494,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:44.012319+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:44.012319+0000","last_peered":"2024-09-18T12:00:44.012319+0000","last_clean":"2024-09-18T12:00:44.012319+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:44.012319+0000","last_undegraded":"2024-09-18T12:00:44.012319+0000","last_fullsized":"2024-09-18T12:00:44.012319+0000","mapping_epoch":40,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25298747,"num_objects":6179,"num_object_clones":0,"num_object_copies":12358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13343,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13206","reported_seq":13289,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.100165+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:43.100165+0000","last_peered":"2024-09-18T12:00:43.100165+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:43.100165+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24544466,"num_objects":5994,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13204,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13307","reported_seq":13394,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.230400+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:43.230400+0000","last_peered":"2024-09-18T12:00:43.230400+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:43.230400+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25418148,"num_objects":6209,"num_object_clones":0,"num_object_copies":12418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13304,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":402858060,"num_objects":98372,"num_object_clones":0,"num_object_copies":196744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98372,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196843,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8823,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":979427328,"data_stored":979356562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051666,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613048,"kb_used_data":311024,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93758792,"statfs":{"total":96636764160,"available":96009003008,"internally_reserved":0,"allocated":318488576,"data_stored":318250460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.248},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.35899999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.21099999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855429,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592900,"kb_used_data":295036,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93778940,"statfs":{"total":96636764160,"available":96029634560,"internally_reserved":0,"allocated":302116864,"data_stored":301822941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":1.5229999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":7.6100000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.41599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655060,"kb_used_data":353096,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93716780,"statfs":{"total":96636764160,"available":95965982720,"internally_reserved":0,"allocated":361570304,"data_stored":361332638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.20899999999999999},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.9359999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":13.773},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.36899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":359960576,"data_stored":359957097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301690880,"data_stored":301631144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317775872,"data_stored":317768321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:45.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:46.723 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:46.729 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:46.745 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-18T12:00:45.178349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":402858079,"num_objects":98374,"num_object_clones":0,"num_object_copies":196748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196845,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8824,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1861008,"kb_used_data":959156,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281254512,"statfs":{"total":289910292480,"available":288004620288,"internally_reserved":0,"allocated":982175744,"data_stored":981406039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,10,4,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26206208,"num_objects":-6398,"num_object_clones":0,"num_object_copies":-12796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-43,"num_objects_misplaced":-352,"num_objects_unfound":0,"num_objects_dirty":-6398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6398,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":395,"num_bytes_recovered":1380352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"2.f","version":"41'13384","reported_seq":13456,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.254169+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:43.254169+0000","last_peered":"2024-09-18T12:00:43.254169+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:43.254169+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25198592,"num_objects":6152,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13384,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2553}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13301","reported_seq":13459,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:43.339220+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:43.339220+0000","last_peered":"2024-09-18T12:00:43.339220+0000","last_clean":"2024-09-18T12:00:43.339220+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:43.339220+0000","last_undegraded":"2024-09-18T12:00:43.339220+0000","last_fullsized":"2024-09-18T12:00:43.339220+0000","mapping_epoch":29,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25112576,"num_objects":6131,"num_object_clones":0,"num_object_copies":12262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13339","reported_seq":13527,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.313970+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:43.313970+0000","last_peered":"2024-09-18T12:00:43.313970+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:43.313970+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25468952,"num_objects":6219,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10197,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2683}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13153","reported_seq":13183,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:43.376522+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:43.376522+0000","last_peered":"2024-09-18T12:00:43.376522+0000","last_clean":"2024-09-18T12:00:43.376522+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:43.376522+0000","last_undegraded":"2024-09-18T12:00:43.376522+0000","last_fullsized":"2024-09-18T12:00:43.376522+0000","mapping_epoch":29,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25014272,"num_objects":6107,"num_object_clones":0,"num_object_copies":12214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13279","reported_seq":13472,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:43.200818+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:43.200818+0000","last_peered":"2024-09-18T12:00:43.200818+0000","last_clean":"2024-09-18T12:00:43.200818+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:43.200818+0000","last_undegraded":"2024-09-18T12:00:43.200818+0000","last_fullsized":"2024-09-18T12:00:43.200818+0000","mapping_epoch":29,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25264128,"num_objects":6168,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10444,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13257","reported_seq":13340,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.195014+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:43.195014+0000","last_peered":"2024-09-18T12:00:43.195014+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:43.195014+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25257170,"num_objects":6168,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13255,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13267","reported_seq":13356,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.248357+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:43.248357+0000","last_peered":"2024-09-18T12:00:43.248357+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:43.248357+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25007314,"num_objects":6107,"num_object_clones":0,"num_object_copies":12214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13265,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13499","reported_seq":13726,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:43.143577+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:43.143577+0000","last_peered":"2024-09-18T12:00:43.143577+0000","last_clean":"2024-09-18T12:00:43.143577+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:43.143577+0000","last_undegraded":"2024-09-18T12:00:43.143577+0000","last_fullsized":"2024-09-18T12:00:43.143577+0000","mapping_epoch":29,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24997888,"num_objects":6103,"num_object_clones":0,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13240","reported_seq":20369,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:44.074421+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:44.074421+0000","last_peered":"2024-09-18T12:00:44.074421+0000","last_clean":"2024-09-18T12:00:44.074421+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:44.074421+0000","last_undegraded":"2024-09-18T12:00:44.074421+0000","last_fullsized":"2024-09-18T12:00:44.074421+0000","mapping_epoch":38,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25253074,"num_objects":6167,"num_object_clones":0,"num_object_copies":12334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13238,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13276","reported_seq":13356,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.179868+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:43.179868+0000","last_peered":"2024-09-18T12:00:43.179868+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:43.179868+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25273554,"num_objects":6172,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13274,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2640}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13311","reported_seq":13369,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.239231+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:43.239231+0000","last_peered":"2024-09-18T12:00:43.239231+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:43.239231+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25199826,"num_objects":6154,"num_object_clones":0,"num_object_copies":12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13309,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2604}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13362","reported_seq":17164,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.360208+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:43.360208+0000","last_peered":"2024-09-18T12:00:43.360208+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:43.360208+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'3300","ondisk_log_start":"30'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25133056,"num_objects":6136,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9842,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1795,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4336}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13323","reported_seq":13405,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.176765+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:43.176765+0000","last_peered":"2024-09-18T12:00:43.176765+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:43.176765+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25416297,"num_objects":6206,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13322,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13346","reported_seq":20494,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:44.012319+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:44.012319+0000","last_peered":"2024-09-18T12:00:44.012319+0000","last_clean":"2024-09-18T12:00:44.012319+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:44.012319+0000","last_undegraded":"2024-09-18T12:00:44.012319+0000","last_fullsized":"2024-09-18T12:00:44.012319+0000","mapping_epoch":40,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25298747,"num_objects":6179,"num_object_clones":0,"num_object_copies":12358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13343,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13206","reported_seq":13289,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.100165+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:43.100165+0000","last_peered":"2024-09-18T12:00:43.100165+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:43.100165+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24544466,"num_objects":5994,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13204,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13307","reported_seq":13394,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.230400+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:43.230400+0000","last_peered":"2024-09-18T12:00:43.230400+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:43.230400+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25418148,"num_objects":6209,"num_object_clones":0,"num_object_copies":12418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13304,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":402858060,"num_objects":98372,"num_object_clones":0,"num_object_copies":196744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98372,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196843,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8823,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":979427328,"data_stored":979356562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051666,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613048,"kb_used_data":311024,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93758792,"statfs":{"total":96636764160,"available":96009003008,"internally_reserved":0,"allocated":318488576,"data_stored":318250460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.248},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.35899999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.21099999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855429,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592900,"kb_used_data":295036,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93778940,"statfs":{"total":96636764160,"available":96029634560,"internally_reserved":0,"allocated":302116864,"data_stored":301822941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":1.5229999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":7.6100000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.41599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655060,"kb_used_data":353096,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93716780,"statfs":{"total":96636764160,"available":95965982720,"internally_reserved":0,"allocated":361570304,"data_stored":361332638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.20899999999999999},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.9359999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":13.773},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.36899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":359960576,"data_stored":359957097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301690880,"data_stored":301631144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317775872,"data_stored":317768321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:46.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:00:46.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'13384", 'reported_seq': 13456, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:43.254169+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:00:43.254169+0000', 'last_peered': '2024-09-18T12:00:43.254169+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:00:43.254169+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "33'3300", 'ondisk_log_start': "33'3300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25198592, 'num_objects': 6152, 'num_object_clones': 0, 'num_object_copies': 12304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13384, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2553}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:00:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'13339", 'reported_seq': 13527, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:43.313970+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:00:43.313970+0000', 'last_peered': '2024-09-18T12:00:43.313970+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:00:43.313970+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "33'3300", 'ondisk_log_start': "33'3300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25468952, 'num_objects': 6219, 'num_object_clones': 0, 'num_object_copies': 12438, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6219, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10197, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2683}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:00:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'13257", 'reported_seq': 13340, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:43.195014+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:00:43.195014+0000', 'last_peered': '2024-09-18T12:00:43.195014+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:00:43.195014+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "30'3200", 'ondisk_log_start': "30'3200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25257170, 'num_objects': 6168, 'num_object_clones': 0, 'num_object_copies': 12336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13255, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2641}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:00:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'13267", 'reported_seq': 13356, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:43.248357+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:00:43.248357+0000', 'last_peered': '2024-09-18T12:00:43.248357+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:00:43.248357+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "30'3200", 'ondisk_log_start': "30'3200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25007314, 'num_objects': 6107, 'num_object_clones': 0, 'num_object_copies': 12214, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13265, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2548}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:00:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'13276", 'reported_seq': 13356, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:43.179868+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:00:43.179868+0000', 'last_peered': '2024-09-18T12:00:43.179868+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:00:43.179868+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "30'3200", 'ondisk_log_start': "30'3200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25273554, 'num_objects': 6172, 'num_object_clones': 0, 'num_object_copies': 12344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13274, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2640}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:00:46.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'13311", 'reported_seq': 13369, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:43.239231+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:00:43.239231+0000', 'last_peered': '2024-09-18T12:00:43.239231+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:00:43.239231+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "33'3300", 'ondisk_log_start': "33'3300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25199826, 'num_objects': 6154, 'num_object_clones': 0, 'num_object_copies': 12308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13309, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2604}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:00:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'13362", 'reported_seq': 17164, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:43.360208+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:00:43.360208+0000', 'last_peered': '2024-09-18T12:00:43.360208+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:00:43.360208+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "30'3300", 'ondisk_log_start': "30'3300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25133056, 'num_objects': 6136, 'num_object_clones': 0, 'num_object_copies': 12272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1800, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9842, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1795, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 4336}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:00:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'13323", 'reported_seq': 13405, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:43.176765+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:00:43.176765+0000', 'last_peered': '2024-09-18T12:00:43.176765+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:00:43.176765+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "33'3300", 'ondisk_log_start': "33'3300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25416297, 'num_objects': 6206, 'num_object_clones': 0, 'num_object_copies': 12412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13322, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2671}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:00:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'13206", 'reported_seq': 13289, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:43.100165+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:00:43.100165+0000', 'last_peered': '2024-09-18T12:00:43.100165+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:00:43.100165+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'3200", 'ondisk_log_start': "30'3200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24544466, 'num_objects': 5994, 'num_object_clones': 0, 'num_object_copies': 11988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13204, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2401}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:00:46.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'13307", 'reported_seq': 13394, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:43.230400+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:00:43.230400+0000', 'last_peered': '2024-09-18T12:00:43.230400+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:00:43.230400+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "33'3300", 'ondisk_log_start': "33'3300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25418148, 'num_objects': 6209, 'num_object_clones': 0, 'num_object_copies': 12418, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6209, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13304, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:46.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:00:47.608 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:47.608 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33769/196748 objects degraded (17.164%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":447,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":98374,"data_bytes":402858079,"bytes_used":1905672192,"bytes_avail":288004620288,"bytes_total":289910292480,"degraded_objects":33769,"degraded_total":196748,"degraded_ratio":0.1716357980767276,"recovering_objects_per_sec":32,"recovering_bytes_per_sec":115004,"recovering_keys_per_sec":0,"num_objects_recovered":395,"num_bytes_recovered":1380352,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":533},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:00:47.627 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:00:47.627 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:48.393 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:48.412 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:48.434 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-09-18T12:00:47.183499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":402858079,"num_objects":98374,"num_object_clones":0,"num_object_copies":196748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196845,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8824,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1861008,"kb_used_data":959156,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281254512,"statfs":{"total":289910292480,"available":288004620288,"internally_reserved":0,"allocated":982175744,"data_stored":981406039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,10,4,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17489920,"num_objects":-4270,"num_object_clones":0,"num_object_copies":-8540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-59,"num_objects_misplaced":-217,"num_objects_unfound":0,"num_objects_dirty":-4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4270,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":892928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007367"},"pg_stats":[{"pgid":"2.f","version":"41'13384","reported_seq":13456,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.254169+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:43.254169+0000","last_peered":"2024-09-18T12:00:43.254169+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:43.254169+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25198592,"num_objects":6152,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13384,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2553}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13301","reported_seq":13459,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:43.339220+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:43.339220+0000","last_peered":"2024-09-18T12:00:43.339220+0000","last_clean":"2024-09-18T12:00:43.339220+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:43.339220+0000","last_undegraded":"2024-09-18T12:00:43.339220+0000","last_fullsized":"2024-09-18T12:00:43.339220+0000","mapping_epoch":29,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25112576,"num_objects":6131,"num_object_clones":0,"num_object_copies":12262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13339","reported_seq":13527,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.313970+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:43.313970+0000","last_peered":"2024-09-18T12:00:43.313970+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:43.313970+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25468952,"num_objects":6219,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10197,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2683}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13153","reported_seq":13183,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:43.376522+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:43.376522+0000","last_peered":"2024-09-18T12:00:43.376522+0000","last_clean":"2024-09-18T12:00:43.376522+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:43.376522+0000","last_undegraded":"2024-09-18T12:00:43.376522+0000","last_fullsized":"2024-09-18T12:00:43.376522+0000","mapping_epoch":29,"log_start":"30'3100","ondisk_log_start":"30'3100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25014272,"num_objects":6107,"num_object_clones":0,"num_object_copies":12214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13279","reported_seq":13472,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:43.200818+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:43.200818+0000","last_peered":"2024-09-18T12:00:43.200818+0000","last_clean":"2024-09-18T12:00:43.200818+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:43.200818+0000","last_undegraded":"2024-09-18T12:00:43.200818+0000","last_fullsized":"2024-09-18T12:00:43.200818+0000","mapping_epoch":29,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25264128,"num_objects":6168,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10444,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13257","reported_seq":13340,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.195014+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:43.195014+0000","last_peered":"2024-09-18T12:00:43.195014+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:43.195014+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25257170,"num_objects":6168,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13255,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13267","reported_seq":13356,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.248357+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:43.248357+0000","last_peered":"2024-09-18T12:00:43.248357+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:43.248357+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25007314,"num_objects":6107,"num_object_clones":0,"num_object_copies":12214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13265,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2548}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13499","reported_seq":13726,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:43.143577+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:43.143577+0000","last_peered":"2024-09-18T12:00:43.143577+0000","last_clean":"2024-09-18T12:00:43.143577+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:43.143577+0000","last_undegraded":"2024-09-18T12:00:43.143577+0000","last_fullsized":"2024-09-18T12:00:43.143577+0000","mapping_epoch":29,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24997888,"num_objects":6103,"num_object_clones":0,"num_object_copies":12206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13240","reported_seq":20369,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:44.074421+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:44.074421+0000","last_peered":"2024-09-18T12:00:44.074421+0000","last_clean":"2024-09-18T12:00:44.074421+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:44.074421+0000","last_undegraded":"2024-09-18T12:00:44.074421+0000","last_fullsized":"2024-09-18T12:00:44.074421+0000","mapping_epoch":38,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25253074,"num_objects":6167,"num_object_clones":0,"num_object_copies":12334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13238,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13276","reported_seq":13356,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.179868+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:43.179868+0000","last_peered":"2024-09-18T12:00:43.179868+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:43.179868+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25273554,"num_objects":6172,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13274,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2640}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13311","reported_seq":13369,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.239231+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:43.239231+0000","last_peered":"2024-09-18T12:00:43.239231+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:43.239231+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25199826,"num_objects":6154,"num_object_clones":0,"num_object_copies":12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13309,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2604}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13362","reported_seq":17164,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.360208+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:43.360208+0000","last_peered":"2024-09-18T12:00:43.360208+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:43.360208+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"30'3300","ondisk_log_start":"30'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25133056,"num_objects":6136,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9842,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1795,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4336}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13323","reported_seq":13405,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.176765+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:43.176765+0000","last_peered":"2024-09-18T12:00:43.176765+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:43.176765+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25416297,"num_objects":6206,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13322,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13346","reported_seq":20494,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:44.012319+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:44.012319+0000","last_peered":"2024-09-18T12:00:44.012319+0000","last_clean":"2024-09-18T12:00:44.012319+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:44.012319+0000","last_undegraded":"2024-09-18T12:00:44.012319+0000","last_fullsized":"2024-09-18T12:00:44.012319+0000","mapping_epoch":40,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25298747,"num_objects":6179,"num_object_clones":0,"num_object_copies":12358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13343,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13206","reported_seq":13289,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.100165+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:43.100165+0000","last_peered":"2024-09-18T12:00:43.100165+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:43.100165+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24544466,"num_objects":5994,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13204,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13307","reported_seq":13394,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:43.230400+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:43.230400+0000","last_peered":"2024-09-18T12:00:43.230400+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:43.230400+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25418148,"num_objects":6209,"num_object_clones":0,"num_object_copies":12418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13304,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":402858060,"num_objects":98372,"num_object_clones":0,"num_object_copies":196744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98372,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196843,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8823,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":979427328,"data_stored":979356562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051666,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613048,"kb_used_data":311024,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93758792,"statfs":{"total":96636764160,"available":96009003008,"internally_reserved":0,"allocated":318488576,"data_stored":318250460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.248},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.35899999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.21099999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855429,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592900,"kb_used_data":295036,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93778940,"statfs":{"total":96636764160,"available":96029634560,"internally_reserved":0,"allocated":302116864,"data_stored":301822941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":1.5229999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":7.6100000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.41599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655060,"kb_used_data":353096,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93716780,"statfs":{"total":96636764160,"available":95965982720,"internally_reserved":0,"allocated":361570304,"data_stored":361332638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.20899999999999999},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.9359999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":13.773},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.36899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":359960576,"data_stored":359957097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301690880,"data_stored":301631144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317775872,"data_stored":317768321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:51.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:52.201 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:52.206 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:52.221 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-18T12:00:51.184447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":394100831,"num_objects":96236,"num_object_clones":0,"num_object_copies":192472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96236,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198983,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8948,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160791,"ondisk_log_size":160791,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1863604,"kb_used_data":940948,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281251916,"statfs":{"total":289910292480,"available":288001961984,"internally_reserved":0,"allocated":963530752,"data_stored":962761047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,4,5,10,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17539072,"num_objects":-4282,"num_object_clones":0,"num_object_copies":-8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":-81,"num_objects_unfound":0,"num_objects_dirty":-4282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4282,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":335872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007407"},"pg_stats":[{"pgid":"2.f","version":"41'13526","reported_seq":13598,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.304029+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:48.304029+0000","last_peered":"2024-09-18T12:00:48.304029+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:48.304029+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"33'3500","ondisk_log_start":"33'3500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24616960,"num_objects":6010,"num_object_clones":0,"num_object_copies":12020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13526,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2411}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13418","reported_seq":13578,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:48.375957+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:48.375957+0000","last_peered":"2024-09-18T12:00:48.375957+0000","last_clean":"2024-09-18T12:00:48.375957+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:48.375957+0000","last_undegraded":"2024-09-18T12:00:48.375957+0000","last_fullsized":"2024-09-18T12:00:48.375957+0000","mapping_epoch":29,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24633344,"num_objects":6014,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10610,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13477","reported_seq":13665,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.248139+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:48.248139+0000","last_peered":"2024-09-18T12:00:48.248139+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:48.248139+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24903704,"num_objects":6081,"num_object_clones":0,"num_object_copies":12162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6081,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10335,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2545}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13280","reported_seq":13310,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:48.405124+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:48.405124+0000","last_peered":"2024-09-18T12:00:48.405124+0000","last_clean":"2024-09-18T12:00:48.405124+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:48.405124+0000","last_undegraded":"2024-09-18T12:00:48.405124+0000","last_fullsized":"2024-09-18T12:00:48.405124+0000","mapping_epoch":29,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24494080,"num_objects":5980,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13280,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13412","reported_seq":13605,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:48.266316+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:48.266316+0000","last_peered":"2024-09-18T12:00:48.266316+0000","last_clean":"2024-09-18T12:00:48.266316+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:48.266316+0000","last_undegraded":"2024-09-18T12:00:48.266316+0000","last_fullsized":"2024-09-18T12:00:48.266316+0000","mapping_epoch":29,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24719360,"num_objects":6035,"num_object_clones":0,"num_object_copies":12070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10577,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13397","reported_seq":13480,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.213603+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:48.213603+0000","last_peered":"2024-09-18T12:00:48.213603+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:48.213603+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24683730,"num_objects":6028,"num_object_clones":0,"num_object_copies":12056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13395,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13399","reported_seq":13484,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.310902+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:48.310902+0000","last_peered":"2024-09-18T12:00:48.310902+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:48.310902+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466642,"num_objects":5975,"num_object_clones":0,"num_object_copies":11950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13397,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13633","reported_seq":13860,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:48.310888+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:48.310888+0000","last_peered":"2024-09-18T12:00:48.310888+0000","last_clean":"2024-09-18T12:00:48.310888+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:48.310888+0000","last_undegraded":"2024-09-18T12:00:48.310888+0000","last_fullsized":"2024-09-18T12:00:48.310888+0000","mapping_epoch":29,"log_start":"33'3600","ondisk_log_start":"33'3600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24449024,"num_objects":5969,"num_object_clones":0,"num_object_copies":11938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13353","reported_seq":20483,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:49.082841+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:49.082841+0000","last_peered":"2024-09-18T12:00:49.082841+0000","last_clean":"2024-09-18T12:00:49.082841+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:49.082841+0000","last_undegraded":"2024-09-18T12:00:49.082841+0000","last_fullsized":"2024-09-18T12:00:49.082841+0000","mapping_epoch":38,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24790226,"num_objects":6054,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13351,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13410","reported_seq":13489,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.308761+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:48.308761+0000","last_peered":"2024-09-18T12:00:48.308761+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:48.308761+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24724690,"num_objects":6038,"num_object_clones":0,"num_object_copies":12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13408,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13451","reported_seq":13509,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.378834+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:48.378834+0000","last_peered":"2024-09-18T12:00:48.378834+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:48.378834+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24626386,"num_objects":6014,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13449,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2464}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13501","reported_seq":17551,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.391836+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:48.391836+0000","last_peered":"2024-09-18T12:00:48.391836+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:48.391836+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"33'3500","ondisk_log_start":"33'3500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24563712,"num_objects":5997,"num_object_clones":0,"num_object_copies":11994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9981,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1919,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13451","reported_seq":13532,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.317454+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:48.317454+0000","last_peered":"2024-09-18T12:00:48.317454+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:48.317454+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24892009,"num_objects":6078,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13450,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2543}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13491","reported_seq":20638,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:49.103765+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:49.103765+0000","last_peered":"2024-09-18T12:00:49.103765+0000","last_clean":"2024-09-18T12:00:49.103765+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:49.103765+0000","last_undegraded":"2024-09-18T12:00:49.103765+0000","last_fullsized":"2024-09-18T12:00:49.103765+0000","mapping_epoch":40,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24704827,"num_objects":6034,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13488,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13348","reported_seq":13431,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.296164+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:48.296164+0000","last_peered":"2024-09-18T12:00:48.296164+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:48.296164+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'3300","ondisk_log_start":"30'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23962834,"num_objects":5852,"num_object_clones":0,"num_object_copies":11704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13346,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2259}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13441","reported_seq":13528,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.343936+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:48.343936+0000","last_peered":"2024-09-18T12:00:48.343936+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:48.343936+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24869284,"num_objects":6075,"num_object_clones":0,"num_object_copies":12150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13438,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2552}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":394100812,"num_objects":96234,"num_object_clones":0,"num_object_copies":192468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198981,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8947,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":961753088,"data_stored":961682322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160788,"ondisk_log_size":160788,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051667,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613124,"kb_used_data":306940,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93758716,"statfs":{"total":96636764160,"available":96008925184,"internally_reserved":0,"allocated":314306560,"data_stored":314068444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.313},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.78500000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.754},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.58599999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855430,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594796,"kb_used_data":288604,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93777044,"statfs":{"total":96636764160,"available":96027693056,"internally_reserved":0,"allocated":295530496,"data_stored":295236573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.186},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.0699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.215}]}]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655684,"kb_used_data":345404,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93716156,"statfs":{"total":96636764160,"available":95965343744,"internally_reserved":0,"allocated":353693696,"data_stored":353456030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,7,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.19400000000000001},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.7679999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.22700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":353284096,"data_stored":353280617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294965248,"data_stored":294905512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313503744,"data_stored":313496193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:52.222 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:52.972 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:52.984 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:52.995 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-18T12:00:51.184447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":394100831,"num_objects":96236,"num_object_clones":0,"num_object_copies":192472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96236,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198983,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8948,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160791,"ondisk_log_size":160791,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1863604,"kb_used_data":940948,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281251916,"statfs":{"total":289910292480,"available":288001961984,"internally_reserved":0,"allocated":963530752,"data_stored":962761047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,4,5,10,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17539072,"num_objects":-4282,"num_object_clones":0,"num_object_copies":-8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":-81,"num_objects_unfound":0,"num_objects_dirty":-4282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4282,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":335872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007407"},"pg_stats":[{"pgid":"2.f","version":"41'13526","reported_seq":13598,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.304029+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:48.304029+0000","last_peered":"2024-09-18T12:00:48.304029+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:48.304029+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"33'3500","ondisk_log_start":"33'3500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24616960,"num_objects":6010,"num_object_clones":0,"num_object_copies":12020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13526,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2411}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13418","reported_seq":13578,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:48.375957+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:48.375957+0000","last_peered":"2024-09-18T12:00:48.375957+0000","last_clean":"2024-09-18T12:00:48.375957+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:48.375957+0000","last_undegraded":"2024-09-18T12:00:48.375957+0000","last_fullsized":"2024-09-18T12:00:48.375957+0000","mapping_epoch":29,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24633344,"num_objects":6014,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10610,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13477","reported_seq":13665,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.248139+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:48.248139+0000","last_peered":"2024-09-18T12:00:48.248139+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:48.248139+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24903704,"num_objects":6081,"num_object_clones":0,"num_object_copies":12162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6081,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10335,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2545}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13280","reported_seq":13310,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:48.405124+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:48.405124+0000","last_peered":"2024-09-18T12:00:48.405124+0000","last_clean":"2024-09-18T12:00:48.405124+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:48.405124+0000","last_undegraded":"2024-09-18T12:00:48.405124+0000","last_fullsized":"2024-09-18T12:00:48.405124+0000","mapping_epoch":29,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24494080,"num_objects":5980,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13280,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13412","reported_seq":13605,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:48.266316+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:48.266316+0000","last_peered":"2024-09-18T12:00:48.266316+0000","last_clean":"2024-09-18T12:00:48.266316+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:48.266316+0000","last_undegraded":"2024-09-18T12:00:48.266316+0000","last_fullsized":"2024-09-18T12:00:48.266316+0000","mapping_epoch":29,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24719360,"num_objects":6035,"num_object_clones":0,"num_object_copies":12070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10577,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13397","reported_seq":13480,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.213603+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:48.213603+0000","last_peered":"2024-09-18T12:00:48.213603+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:48.213603+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24683730,"num_objects":6028,"num_object_clones":0,"num_object_copies":12056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13395,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13399","reported_seq":13484,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.310902+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:48.310902+0000","last_peered":"2024-09-18T12:00:48.310902+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:48.310902+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466642,"num_objects":5975,"num_object_clones":0,"num_object_copies":11950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13397,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13633","reported_seq":13860,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:48.310888+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:48.310888+0000","last_peered":"2024-09-18T12:00:48.310888+0000","last_clean":"2024-09-18T12:00:48.310888+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:48.310888+0000","last_undegraded":"2024-09-18T12:00:48.310888+0000","last_fullsized":"2024-09-18T12:00:48.310888+0000","mapping_epoch":29,"log_start":"33'3600","ondisk_log_start":"33'3600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24449024,"num_objects":5969,"num_object_clones":0,"num_object_copies":11938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13353","reported_seq":20483,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:49.082841+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:49.082841+0000","last_peered":"2024-09-18T12:00:49.082841+0000","last_clean":"2024-09-18T12:00:49.082841+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:49.082841+0000","last_undegraded":"2024-09-18T12:00:49.082841+0000","last_fullsized":"2024-09-18T12:00:49.082841+0000","mapping_epoch":38,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24790226,"num_objects":6054,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13351,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13410","reported_seq":13489,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.308761+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:48.308761+0000","last_peered":"2024-09-18T12:00:48.308761+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:48.308761+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24724690,"num_objects":6038,"num_object_clones":0,"num_object_copies":12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13408,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13451","reported_seq":13509,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.378834+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:48.378834+0000","last_peered":"2024-09-18T12:00:48.378834+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:48.378834+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24626386,"num_objects":6014,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13449,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2464}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13501","reported_seq":17551,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.391836+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:48.391836+0000","last_peered":"2024-09-18T12:00:48.391836+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:48.391836+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"33'3500","ondisk_log_start":"33'3500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24563712,"num_objects":5997,"num_object_clones":0,"num_object_copies":11994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9981,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1919,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13451","reported_seq":13532,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.317454+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:48.317454+0000","last_peered":"2024-09-18T12:00:48.317454+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:48.317454+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24892009,"num_objects":6078,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13450,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2543}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13491","reported_seq":20638,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:49.103765+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:49.103765+0000","last_peered":"2024-09-18T12:00:49.103765+0000","last_clean":"2024-09-18T12:00:49.103765+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:49.103765+0000","last_undegraded":"2024-09-18T12:00:49.103765+0000","last_fullsized":"2024-09-18T12:00:49.103765+0000","mapping_epoch":40,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24704827,"num_objects":6034,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13488,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13348","reported_seq":13431,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.296164+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:48.296164+0000","last_peered":"2024-09-18T12:00:48.296164+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:48.296164+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'3300","ondisk_log_start":"30'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23962834,"num_objects":5852,"num_object_clones":0,"num_object_copies":11704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13346,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2259}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13441","reported_seq":13528,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.343936+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:48.343936+0000","last_peered":"2024-09-18T12:00:48.343936+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:48.343936+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24869284,"num_objects":6075,"num_object_clones":0,"num_object_copies":12150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13438,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2552}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":394100812,"num_objects":96234,"num_object_clones":0,"num_object_copies":192468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198981,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8947,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":961753088,"data_stored":961682322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160788,"ondisk_log_size":160788,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051667,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613124,"kb_used_data":306940,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93758716,"statfs":{"total":96636764160,"available":96008925184,"internally_reserved":0,"allocated":314306560,"data_stored":314068444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.313},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.78500000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.754},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.58599999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855430,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594796,"kb_used_data":288604,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93777044,"statfs":{"total":96636764160,"available":96027693056,"internally_reserved":0,"allocated":295530496,"data_stored":295236573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.186},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.0699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.215}]}]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655684,"kb_used_data":345404,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93716156,"statfs":{"total":96636764160,"available":95965343744,"internally_reserved":0,"allocated":353693696,"data_stored":353456030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,7,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.19400000000000001},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.7679999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.22700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":353284096,"data_stored":353280617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294965248,"data_stored":294905512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313503744,"data_stored":313496193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:52.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:00:52.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'13526", 'reported_seq': 13598, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:48.304029+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:00:48.304029+0000', 'last_peered': '2024-09-18T12:00:48.304029+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:00:48.304029+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "33'3500", 'ondisk_log_start': "33'3500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24616960, 'num_objects': 6010, 'num_object_clones': 0, 'num_object_copies': 12020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13526, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2411}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:52.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:00:52.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'13477", 'reported_seq': 13665, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:48.248139+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:00:48.248139+0000', 'last_peered': '2024-09-18T12:00:48.248139+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:00:48.248139+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "33'3400", 'ondisk_log_start': "33'3400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24903704, 'num_objects': 6081, 'num_object_clones': 0, 'num_object_copies': 12162, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6081, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10335, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2545}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:52.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:00:52.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'13397", 'reported_seq': 13480, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:48.213603+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:00:48.213603+0000', 'last_peered': '2024-09-18T12:00:48.213603+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:00:48.213603+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "33'3300", 'ondisk_log_start': "33'3300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24683730, 'num_objects': 6028, 'num_object_clones': 0, 'num_object_copies': 12056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6028, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13395, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:52.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:00:52.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'13399", 'reported_seq': 13484, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:48.310902+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:00:48.310902+0000', 'last_peered': '2024-09-18T12:00:48.310902+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:00:48.310902+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "33'3300", 'ondisk_log_start': "33'3300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24466642, 'num_objects': 5975, 'num_object_clones': 0, 'num_object_copies': 11950, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13397, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2416}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:52.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:00:52.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'13410", 'reported_seq': 13489, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:48.308761+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:00:48.308761+0000', 'last_peered': '2024-09-18T12:00:48.308761+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:00:48.308761+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "33'3400", 'ondisk_log_start': "33'3400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24724690, 'num_objects': 6038, 'num_object_clones': 0, 'num_object_copies': 12076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13408, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2506}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:52.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:00:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'13451", 'reported_seq': 13509, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:48.378834+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:00:48.378834+0000', 'last_peered': '2024-09-18T12:00:48.378834+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:00:48.378834+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "33'3400", 'ondisk_log_start': "33'3400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24626386, 'num_objects': 6014, 'num_object_clones': 0, 'num_object_copies': 12028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13449, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2464}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:00:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'13501", 'reported_seq': 17551, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:48.391836+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:00:48.391836+0000', 'last_peered': '2024-09-18T12:00:48.391836+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:00:48.391836+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "33'3500", 'ondisk_log_start': "33'3500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24563712, 'num_objects': 5997, 'num_object_clones': 0, 'num_object_copies': 11994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1676, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9981, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1919, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 4321}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:00:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'13451", 'reported_seq': 13532, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:48.317454+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:00:48.317454+0000', 'last_peered': '2024-09-18T12:00:48.317454+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:00:48.317454+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "33'3400", 'ondisk_log_start': "33'3400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24892009, 'num_objects': 6078, 'num_object_clones': 0, 'num_object_copies': 12156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6078, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13450, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2543}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:00:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'13348", 'reported_seq': 13431, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:48.296164+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:00:48.296164+0000', 'last_peered': '2024-09-18T12:00:48.296164+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:00:48.296164+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "30'3300", 'ondisk_log_start': "30'3300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23962834, 'num_objects': 5852, 'num_object_clones': 0, 'num_object_copies': 11704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13346, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2259}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:00:52.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'13441", 'reported_seq': 13528, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:48.343936+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:00:48.343936+0000', 'last_peered': '2024-09-18T12:00:48.343936+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:00:48.343936+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "33'3400", 'ondisk_log_start': "33'3400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24869284, 'num_objects': 6075, 'num_object_clones': 0, 'num_object_copies': 12150, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13438, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:52.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:00:53.868 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:53.868 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33645/192472 objects degraded (17.480%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":453,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":96236,"data_bytes":394100831,"bytes_used":1908330496,"bytes_avail":288001961984,"bytes_total":289910292480,"degraded_objects":33645,"degraded_total":192472,"degraded_ratio":0.17480464690968037,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":27966,"recovering_keys_per_sec":0,"num_objects_recovered":264,"num_bytes_recovered":335872,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":356},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:00:53.905 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:00:53.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:54.806 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:54.819 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-18T12:00:53.187509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":394100831,"num_objects":96236,"num_object_clones":0,"num_object_copies":192472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96236,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198983,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8948,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160791,"ondisk_log_size":160791,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1863604,"kb_used_data":940948,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281251916,"statfs":{"total":289910292480,"available":288001961984,"internally_reserved":0,"allocated":963530752,"data_stored":962761047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,4,5,10,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17539072,"num_objects":-4282,"num_object_clones":0,"num_object_copies":-8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":-81,"num_objects_unfound":0,"num_objects_dirty":-4282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4282,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":335872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009979"},"pg_stats":[{"pgid":"2.f","version":"41'13526","reported_seq":13598,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.304029+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:48.304029+0000","last_peered":"2024-09-18T12:00:48.304029+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:48.304029+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"33'3500","ondisk_log_start":"33'3500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24616960,"num_objects":6010,"num_object_clones":0,"num_object_copies":12020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13526,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2411}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13418","reported_seq":13578,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:48.375957+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:48.375957+0000","last_peered":"2024-09-18T12:00:48.375957+0000","last_clean":"2024-09-18T12:00:48.375957+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:48.375957+0000","last_undegraded":"2024-09-18T12:00:48.375957+0000","last_fullsized":"2024-09-18T12:00:48.375957+0000","mapping_epoch":29,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24633344,"num_objects":6014,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10610,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13477","reported_seq":13665,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.248139+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:48.248139+0000","last_peered":"2024-09-18T12:00:48.248139+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:48.248139+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24903704,"num_objects":6081,"num_object_clones":0,"num_object_copies":12162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6081,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10335,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2545}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13280","reported_seq":13310,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:48.405124+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:48.405124+0000","last_peered":"2024-09-18T12:00:48.405124+0000","last_clean":"2024-09-18T12:00:48.405124+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:48.405124+0000","last_undegraded":"2024-09-18T12:00:48.405124+0000","last_fullsized":"2024-09-18T12:00:48.405124+0000","mapping_epoch":29,"log_start":"30'3200","ondisk_log_start":"30'3200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24494080,"num_objects":5980,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13280,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13412","reported_seq":13605,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:48.266316+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:48.266316+0000","last_peered":"2024-09-18T12:00:48.266316+0000","last_clean":"2024-09-18T12:00:48.266316+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:48.266316+0000","last_undegraded":"2024-09-18T12:00:48.266316+0000","last_fullsized":"2024-09-18T12:00:48.266316+0000","mapping_epoch":29,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24719360,"num_objects":6035,"num_object_clones":0,"num_object_copies":12070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10577,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13397","reported_seq":13480,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.213603+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:48.213603+0000","last_peered":"2024-09-18T12:00:48.213603+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:48.213603+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24683730,"num_objects":6028,"num_object_clones":0,"num_object_copies":12056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13395,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13399","reported_seq":13484,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.310902+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:48.310902+0000","last_peered":"2024-09-18T12:00:48.310902+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:48.310902+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466642,"num_objects":5975,"num_object_clones":0,"num_object_copies":11950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13397,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2416}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13633","reported_seq":13860,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:48.310888+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:48.310888+0000","last_peered":"2024-09-18T12:00:48.310888+0000","last_clean":"2024-09-18T12:00:48.310888+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:48.310888+0000","last_undegraded":"2024-09-18T12:00:48.310888+0000","last_fullsized":"2024-09-18T12:00:48.310888+0000","mapping_epoch":29,"log_start":"33'3600","ondisk_log_start":"33'3600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24449024,"num_objects":5969,"num_object_clones":0,"num_object_copies":11938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13353","reported_seq":20483,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:49.082841+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:49.082841+0000","last_peered":"2024-09-18T12:00:49.082841+0000","last_clean":"2024-09-18T12:00:49.082841+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:49.082841+0000","last_undegraded":"2024-09-18T12:00:49.082841+0000","last_fullsized":"2024-09-18T12:00:49.082841+0000","mapping_epoch":38,"log_start":"33'3300","ondisk_log_start":"33'3300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24790226,"num_objects":6054,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13351,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13410","reported_seq":13489,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.308761+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:48.308761+0000","last_peered":"2024-09-18T12:00:48.308761+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:48.308761+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24724690,"num_objects":6038,"num_object_clones":0,"num_object_copies":12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13408,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13451","reported_seq":13509,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.378834+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:48.378834+0000","last_peered":"2024-09-18T12:00:48.378834+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:48.378834+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24626386,"num_objects":6014,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13449,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2464}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13501","reported_seq":17551,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.391836+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:48.391836+0000","last_peered":"2024-09-18T12:00:48.391836+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:48.391836+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"33'3500","ondisk_log_start":"33'3500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24563712,"num_objects":5997,"num_object_clones":0,"num_object_copies":11994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9981,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1919,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13451","reported_seq":13532,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.317454+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:48.317454+0000","last_peered":"2024-09-18T12:00:48.317454+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:48.317454+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24892009,"num_objects":6078,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13450,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2543}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13491","reported_seq":20638,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:49.103765+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:49.103765+0000","last_peered":"2024-09-18T12:00:49.103765+0000","last_clean":"2024-09-18T12:00:49.103765+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:49.103765+0000","last_undegraded":"2024-09-18T12:00:49.103765+0000","last_fullsized":"2024-09-18T12:00:49.103765+0000","mapping_epoch":40,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24704827,"num_objects":6034,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13488,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13348","reported_seq":13431,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.296164+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:48.296164+0000","last_peered":"2024-09-18T12:00:48.296164+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:48.296164+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"30'3300","ondisk_log_start":"30'3300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23962834,"num_objects":5852,"num_object_clones":0,"num_object_copies":11704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13346,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2259}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13441","reported_seq":13528,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:48.343936+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:48.343936+0000","last_peered":"2024-09-18T12:00:48.343936+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:48.343936+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24869284,"num_objects":6075,"num_object_clones":0,"num_object_copies":12150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13438,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2552}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":394100812,"num_objects":96234,"num_object_clones":0,"num_object_copies":192468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198981,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8947,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":961753088,"data_stored":961682322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160788,"ondisk_log_size":160788,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051667,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613124,"kb_used_data":306940,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93758716,"statfs":{"total":96636764160,"available":96008925184,"internally_reserved":0,"allocated":314306560,"data_stored":314068444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.313},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.78500000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.754},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.58599999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855430,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594796,"kb_used_data":288604,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93777044,"statfs":{"total":96636764160,"available":96027693056,"internally_reserved":0,"allocated":295530496,"data_stored":295236573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.186},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.0699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.215}]}]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655684,"kb_used_data":345404,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93716156,"statfs":{"total":96636764160,"available":95965343744,"internally_reserved":0,"allocated":353693696,"data_stored":353456030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,7,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.19400000000000001},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.7679999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.22700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":353284096,"data_stored":353280617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294965248,"data_stored":294905512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313503744,"data_stored":313496193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:57.822 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:58.579 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:58.605 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:58.624 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-18T12:00:57.188473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":385290335,"num_objects":94085,"num_object_clones":0,"num_object_copies":188170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94085,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201134,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9086,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160742,"ondisk_log_size":160742,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1842056,"kb_used_data":923564,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281273464,"statfs":{"total":289910292480,"available":288024027136,"internally_reserved":0,"allocated":945729536,"data_stored":944959831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,3,7,10,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17567744,"num_objects":-4289,"num_object_clones":0,"num_object_copies":-8578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4289,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010124"},"pg_stats":[{"pgid":"2.f","version":"41'13649","reported_seq":13721,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:53.126372+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:53.126372+0000","last_peered":"2024-09-18T12:00:53.126372+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:53.126372+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24113152,"num_objects":5887,"num_object_clones":0,"num_object_copies":11774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13649,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2288}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13566","reported_seq":13726,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:53.270784+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:53.270784+0000","last_peered":"2024-09-18T12:00:53.270784+0000","last_clean":"2024-09-18T12:00:53.270784+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:53.270784+0000","last_undegraded":"2024-09-18T12:00:53.270784+0000","last_fullsized":"2024-09-18T12:00:53.270784+0000","mapping_epoch":29,"log_start":"34'3500","ondisk_log_start":"34'3500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24027136,"num_objects":5866,"num_object_clones":0,"num_object_copies":11732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10758,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13604","reported_seq":13792,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:53.404994+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:53.404994+0000","last_peered":"2024-09-18T12:00:53.404994+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:53.404994+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24383512,"num_objects":5954,"num_object_clones":0,"num_object_copies":11908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10462,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2418}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13407","reported_seq":13438,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:53.301350+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:53.301350+0000","last_peered":"2024-09-18T12:00:53.301350+0000","last_clean":"2024-09-18T12:00:53.301350+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:53.301350+0000","last_undegraded":"2024-09-18T12:00:53.301350+0000","last_fullsized":"2024-09-18T12:00:53.301350+0000","mapping_epoch":29,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23973888,"num_objects":5853,"num_object_clones":0,"num_object_copies":11706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13407,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13527","reported_seq":13720,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:53.312624+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:53.312624+0000","last_peered":"2024-09-18T12:00:53.312624+0000","last_clean":"2024-09-18T12:00:53.312624+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:53.312624+0000","last_undegraded":"2024-09-18T12:00:53.312624+0000","last_fullsized":"2024-09-18T12:00:53.312624+0000","mapping_epoch":29,"log_start":"34'3500","ondisk_log_start":"34'3500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24248320,"num_objects":5920,"num_object_clones":0,"num_object_copies":11840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10692,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13537","reported_seq":13620,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:53.335510+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:53.335510+0000","last_peered":"2024-09-18T12:00:53.335510+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:53.335510+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"34'3500","ondisk_log_start":"34'3500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24110290,"num_objects":5888,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13535,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2361}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13535","reported_seq":13622,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:53.353033+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:53.353033+0000","last_peered":"2024-09-18T12:00:53.353033+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:53.353033+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"34'3500","ondisk_log_start":"34'3500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23909586,"num_objects":5839,"num_object_clones":0,"num_object_copies":11678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13533,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13766","reported_seq":13993,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:53.282041+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:53.282041+0000","last_peered":"2024-09-18T12:00:53.282041+0000","last_clean":"2024-09-18T12:00:53.282041+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:53.282041+0000","last_undegraded":"2024-09-18T12:00:53.282041+0000","last_fullsized":"2024-09-18T12:00:53.282041+0000","mapping_epoch":29,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23904256,"num_objects":5836,"num_object_clones":0,"num_object_copies":11672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10083,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13482","reported_seq":20612,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:54.202803+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:54.202803+0000","last_peered":"2024-09-18T12:00:54.202803+0000","last_clean":"2024-09-18T12:00:54.202803+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:54.202803+0000","last_undegraded":"2024-09-18T12:00:54.202803+0000","last_fullsized":"2024-09-18T12:00:54.202803+0000","mapping_epoch":38,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24261842,"num_objects":5925,"num_object_clones":0,"num_object_copies":11850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13480,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13536","reported_seq":13616,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:53.342299+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:53.342299+0000","last_peered":"2024-09-18T12:00:53.342299+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:53.342299+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"34'3500","ondisk_log_start":"34'3500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24208594,"num_objects":5912,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13534,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2380}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13576","reported_seq":13634,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:53.395322+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:53.395322+0000","last_peered":"2024-09-18T12:00:53.395322+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:53.395322+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"34'3500","ondisk_log_start":"34'3500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114386,"num_objects":5889,"num_object_clones":0,"num_object_copies":11778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13574,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13641","reported_seq":17967,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:53.463695+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:53.463695+0000","last_peered":"2024-09-18T12:00:53.463695+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:53.463695+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23990272,"num_objects":5857,"num_object_clones":0,"num_object_copies":11714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10121,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2057,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13603","reported_seq":13687,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:53.244653+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:53.244653+0000","last_peered":"2024-09-18T12:00:53.244653+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:53.244653+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24269417,"num_objects":5926,"num_object_clones":0,"num_object_copies":11852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13602,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2391}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13634","reported_seq":20782,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:54.197592+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:54.197592+0000","last_peered":"2024-09-18T12:00:54.197592+0000","last_clean":"2024-09-18T12:00:54.197592+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:54.197592+0000","last_undegraded":"2024-09-18T12:00:54.197592+0000","last_fullsized":"2024-09-18T12:00:54.197592+0000","mapping_epoch":40,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24119099,"num_objects":5891,"num_object_clones":0,"num_object_copies":11782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13631,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13489","reported_seq":13572,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:53.252935+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:53.252935+0000","last_peered":"2024-09-18T12:00:53.252935+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:53.252935+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23385298,"num_objects":5711,"num_object_clones":0,"num_object_copies":11422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13487,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2118}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13587","reported_seq":13674,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:53.206414+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:53.206414+0000","last_peered":"2024-09-18T12:00:53.206414+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:53.206414+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"34'3500","ondisk_log_start":"34'3500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24271268,"num_objects":5929,"num_object_clones":0,"num_object_copies":11858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13584,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":385290316,"num_objects":94083,"num_object_clones":0,"num_object_copies":188166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94083,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201132,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9085,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":944119808,"data_stored":944049042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051668,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":612996,"kb_used_data":302652,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93758844,"statfs":{"total":96636764160,"available":96009056256,"internally_reserved":0,"allocated":309915648,"data_stored":309677532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":2.4009999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":2.4830000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.30399999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855431,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592460,"kb_used_data":282112,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93779380,"statfs":{"total":96636764160,"available":96030085120,"internally_reserved":0,"allocated":288882688,"data_stored":288588765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.186},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.0699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.215}]}]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636600,"kb_used_data":338800,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93735240,"statfs":{"total":96636764160,"available":95984885760,"internally_reserved":0,"allocated":346931200,"data_stored":346693534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,7,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.19}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":4.0499999999999998},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346558464,"data_stored":346554985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288333824,"data_stored":288274088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":309227520,"data_stored":309219969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:58.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:00:59.422 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:00:59.448 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:00:59.489 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-09-18T12:00:59.188898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":377409631,"num_objects":92161,"num_object_clones":0,"num_object_copies":184322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92161,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203058,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9218,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1823896,"kb_used_data":913724,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281291624,"statfs":{"total":289910292480,"available":288042622976,"internally_reserved":0,"allocated":935653376,"data_stored":934883671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,8,9,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25448448,"num_objects":-6213,"num_object_clones":0,"num_object_copies":-12426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":394,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005398"},"pg_stats":[{"pgid":"2.f","version":"41'13788","reported_seq":13860,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.364957+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:58.364957+0000","last_peered":"2024-09-18T12:00:58.364957+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:58.364957+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23543808,"num_objects":5748,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13788,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2149}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13727","reported_seq":13887,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:58.425458+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:58.425458+0000","last_peered":"2024-09-18T12:00:58.425458+0000","last_clean":"2024-09-18T12:00:58.425458+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:58.425458+0000","last_undegraded":"2024-09-18T12:00:58.425458+0000","last_fullsized":"2024-09-18T12:00:58.425458+0000","mapping_epoch":29,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23367680,"num_objects":5705,"num_object_clones":0,"num_object_copies":11410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10919,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13745","reported_seq":13933,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.338329+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:58.338329+0000","last_peered":"2024-09-18T12:00:58.338329+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:58.338329+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23805976,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10603,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2277}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13550","reported_seq":13581,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:58.377357+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:58.377357+0000","last_peered":"2024-09-18T12:00:58.377357+0000","last_clean":"2024-09-18T12:00:58.377357+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:58.377357+0000","last_undegraded":"2024-09-18T12:00:58.377357+0000","last_fullsized":"2024-09-18T12:00:58.377357+0000","mapping_epoch":29,"log_start":"34'3500","ondisk_log_start":"34'3500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23388160,"num_objects":5710,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13550,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13669","reported_seq":13862,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:58.276327+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:58.276327+0000","last_peered":"2024-09-18T12:00:58.276327+0000","last_clean":"2024-09-18T12:00:58.276327+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:58.276327+0000","last_undegraded":"2024-09-18T12:00:58.276327+0000","last_fullsized":"2024-09-18T12:00:58.276327+0000","mapping_epoch":29,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23666688,"num_objects":5778,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10834,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13661","reported_seq":13744,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.307629+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:58.307629+0000","last_peered":"2024-09-18T12:00:58.307629+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:58.307629+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23602386,"num_objects":5764,"num_object_clones":0,"num_object_copies":11528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13659,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13662","reported_seq":13751,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.384882+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:58.384882+0000","last_peered":"2024-09-18T12:00:58.384882+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:58.384882+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23389394,"num_objects":5712,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13660,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13901","reported_seq":14128,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:58.344879+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:58.344879+0000","last_peered":"2024-09-18T12:00:58.344879+0000","last_clean":"2024-09-18T12:00:58.344879+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:58.344879+0000","last_undegraded":"2024-09-18T12:00:58.344879+0000","last_fullsized":"2024-09-18T12:00:58.344879+0000","mapping_epoch":29,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23351296,"num_objects":5701,"num_object_clones":0,"num_object_copies":11402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10218,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13482","reported_seq":20612,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:54.202803+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:54.202803+0000","last_peered":"2024-09-18T12:00:54.202803+0000","last_clean":"2024-09-18T12:00:54.202803+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:54.202803+0000","last_undegraded":"2024-09-18T12:00:54.202803+0000","last_fullsized":"2024-09-18T12:00:54.202803+0000","mapping_epoch":38,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24261842,"num_objects":5925,"num_object_clones":0,"num_object_copies":11850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13480,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13652","reported_seq":13732,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.252678+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:58.252678+0000","last_peered":"2024-09-18T12:00:58.252678+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:58.252678+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23733458,"num_objects":5796,"num_object_clones":0,"num_object_copies":11592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13650,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2264}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13720","reported_seq":13778,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.385841+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:58.385841+0000","last_peered":"2024-09-18T12:00:58.385841+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:58.385841+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23524562,"num_objects":5745,"num_object_clones":0,"num_object_copies":11490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13718,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13776","reported_seq":18366,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.466090+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:58.466090+0000","last_peered":"2024-09-18T12:00:58.466090+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:58.466090+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23437312,"num_objects":5722,"num_object_clones":0,"num_object_copies":11444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2189,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13757","reported_seq":13840,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.390415+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:58.390415+0000","last_peered":"2024-09-18T12:00:58.390415+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:58.390415+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23638633,"num_objects":5772,"num_object_clones":0,"num_object_copies":11544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13756,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13634","reported_seq":20782,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:54.197592+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:54.197592+0000","last_peered":"2024-09-18T12:00:54.197592+0000","last_clean":"2024-09-18T12:00:54.197592+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:54.197592+0000","last_undegraded":"2024-09-18T12:00:54.197592+0000","last_fullsized":"2024-09-18T12:00:54.197592+0000","mapping_epoch":40,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24119099,"num_objects":5891,"num_object_clones":0,"num_object_copies":11782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13631,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13614","reported_seq":13697,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.340785+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:58.340785+0000","last_peered":"2024-09-18T12:00:58.340785+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:58.340785+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22873298,"num_objects":5586,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13612,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13725","reported_seq":13812,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.315852+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:58.315852+0000","last_peered":"2024-09-18T12:00:58.315852+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:58.315852+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23706020,"num_objects":5791,"num_object_clones":0,"num_object_copies":11582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13722,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2268}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":377409612,"num_objects":92159,"num_object_clones":0,"num_object_copies":184318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203056,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9217,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":932843520,"data_stored":932772754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160763,"ondisk_log_size":160763,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051669,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595816,"kb_used_data":297952,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93776024,"statfs":{"total":96636764160,"available":96026648576,"internally_reserved":0,"allocated":305102848,"data_stored":304864732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":7.1740000000000004},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.36599999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":7.2539999999999996}]}]},{"osd":1,"up_from":35,"seq":150323855431,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592460,"kb_used_data":282112,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93779380,"statfs":{"total":96636764160,"available":96030085120,"internally_reserved":0,"allocated":288882688,"data_stored":288588765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.186},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.0699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.215}]}]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635620,"kb_used_data":333660,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93736220,"statfs":{"total":96636764160,"available":95985889280,"internally_reserved":0,"allocated":341667840,"data_stored":341430174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339927040,"data_stored":339923561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288333824,"data_stored":288274088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304582656,"data_stored":304575105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:00:59.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:00:59.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'13788", 'reported_seq': 13860, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:58.364957+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:00:58.364957+0000', 'last_peered': '2024-09-18T12:00:58.364957+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:00:58.364957+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'3700", 'ondisk_log_start': "37'3700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23543808, 'num_objects': 5748, 'num_object_clones': 0, 'num_object_copies': 11496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13788, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2149}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:59.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:00:59.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'13745", 'reported_seq': 13933, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:58.338329+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:00:58.338329+0000', 'last_peered': '2024-09-18T12:00:58.338329+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:00:58.338329+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'3700", 'ondisk_log_start': "37'3700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23805976, 'num_objects': 5813, 'num_object_clones': 0, 'num_object_copies': 11626, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5813, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10603, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:59.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:00:59.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'13661", 'reported_seq': 13744, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:58.307629+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:00:58.307629+0000', 'last_peered': '2024-09-18T12:00:58.307629+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:00:58.307629+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'3600", 'ondisk_log_start': "37'3600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23602386, 'num_objects': 5764, 'num_object_clones': 0, 'num_object_copies': 11528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13659, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:59.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:00:59.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'13662", 'reported_seq': 13751, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:58.384882+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:00:58.384882+0000', 'last_peered': '2024-09-18T12:00:58.384882+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:00:58.384882+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'3600", 'ondisk_log_start': "37'3600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23389394, 'num_objects': 5712, 'num_object_clones': 0, 'num_object_copies': 11424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5712, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13660, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2153}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:59.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:00:59.491 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'13652", 'reported_seq': 13732, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:58.252678+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:00:58.252678+0000', 'last_peered': '2024-09-18T12:00:58.252678+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:00:58.252678+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'3600", 'ondisk_log_start': "37'3600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23733458, 'num_objects': 5796, 'num_object_clones': 0, 'num_object_copies': 11592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13650, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2264}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:00:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'13720", 'reported_seq': 13778, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:58.385841+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:00:58.385841+0000', 'last_peered': '2024-09-18T12:00:58.385841+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:00:58.385841+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'3700", 'ondisk_log_start': "37'3700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23524562, 'num_objects': 5745, 'num_object_clones': 0, 'num_object_copies': 11490, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5745, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13718, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2195}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:00:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'13776", 'reported_seq': 18366, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:58.466090+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:00:58.466090+0000', 'last_peered': '2024-09-18T12:00:58.466090+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:00:58.466090+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "37'3700", 'ondisk_log_start': "37'3700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23437312, 'num_objects': 5722, 'num_object_clones': 0, 'num_object_copies': 11444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1406, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10256, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2189, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 4316}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:00:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:00:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'13757", 'reported_seq': 13840, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:58.390415+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:00:58.390415+0000', 'last_peered': '2024-09-18T12:00:58.390415+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:00:58.390415+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'3700", 'ondisk_log_start': "37'3700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23638633, 'num_objects': 5772, 'num_object_clones': 0, 'num_object_copies': 11544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13756, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2237}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:00:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'13614", 'reported_seq': 13697, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:58.340785+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:00:58.340785+0000', 'last_peered': '2024-09-18T12:00:58.340785+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:00:58.340785+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'3600", 'ondisk_log_start': "37'3600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22873298, 'num_objects': 5586, 'num_object_clones': 0, 'num_object_copies': 11172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13612, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1993}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:00:59.492 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'13725", 'reported_seq': 13812, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:00:58.315852+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:00:58.315852+0000', 'last_peered': '2024-09-18T12:00:58.315852+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:00:58.315852+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'3700", 'ondisk_log_start': "37'3700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23706020, 'num_objects': 5791, 'num_object_clones': 0, 'num_object_copies': 11582, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13722, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2268}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:00:59.493 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:01:00.296 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:00.297 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33375/184322 objects degraded (18.107%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":460,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":92161,"data_bytes":377409631,"bytes_used":1867669504,"bytes_avail":288042622976,"bytes_total":289910292480,"degraded_objects":33375,"degraded_total":184322,"degraded_ratio":0.18106899881728716,"recovering_objects_per_sec":32,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":394,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":517},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:01:00.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:01:00.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:01.029 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:01.054 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:01.057 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-09-18T12:00:59.188898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":377409631,"num_objects":92161,"num_object_clones":0,"num_object_copies":184322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92161,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203058,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9218,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1823896,"kb_used_data":913724,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281291624,"statfs":{"total":289910292480,"available":288042622976,"internally_reserved":0,"allocated":935653376,"data_stored":934883671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,4,8,9,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25448448,"num_objects":-6213,"num_object_clones":0,"num_object_copies":-12426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":394,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005398"},"pg_stats":[{"pgid":"2.f","version":"41'13788","reported_seq":13860,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.364957+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:58.364957+0000","last_peered":"2024-09-18T12:00:58.364957+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:58.364957+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23543808,"num_objects":5748,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13788,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2149}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13727","reported_seq":13887,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:58.425458+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:58.425458+0000","last_peered":"2024-09-18T12:00:58.425458+0000","last_clean":"2024-09-18T12:00:58.425458+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:58.425458+0000","last_undegraded":"2024-09-18T12:00:58.425458+0000","last_fullsized":"2024-09-18T12:00:58.425458+0000","mapping_epoch":29,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23367680,"num_objects":5705,"num_object_clones":0,"num_object_copies":11410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10919,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13745","reported_seq":13933,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.338329+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:58.338329+0000","last_peered":"2024-09-18T12:00:58.338329+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:58.338329+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23805976,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10603,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2277}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13550","reported_seq":13581,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:58.377357+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:58.377357+0000","last_peered":"2024-09-18T12:00:58.377357+0000","last_clean":"2024-09-18T12:00:58.377357+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:58.377357+0000","last_undegraded":"2024-09-18T12:00:58.377357+0000","last_fullsized":"2024-09-18T12:00:58.377357+0000","mapping_epoch":29,"log_start":"34'3500","ondisk_log_start":"34'3500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23388160,"num_objects":5710,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13550,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13669","reported_seq":13862,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:58.276327+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:58.276327+0000","last_peered":"2024-09-18T12:00:58.276327+0000","last_clean":"2024-09-18T12:00:58.276327+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:58.276327+0000","last_undegraded":"2024-09-18T12:00:58.276327+0000","last_fullsized":"2024-09-18T12:00:58.276327+0000","mapping_epoch":29,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23666688,"num_objects":5778,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10834,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13661","reported_seq":13744,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.307629+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:58.307629+0000","last_peered":"2024-09-18T12:00:58.307629+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:58.307629+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23602386,"num_objects":5764,"num_object_clones":0,"num_object_copies":11528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13659,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13662","reported_seq":13751,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.384882+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:58.384882+0000","last_peered":"2024-09-18T12:00:58.384882+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:58.384882+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23389394,"num_objects":5712,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13660,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13901","reported_seq":14128,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:58.344879+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:58.344879+0000","last_peered":"2024-09-18T12:00:58.344879+0000","last_clean":"2024-09-18T12:00:58.344879+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:58.344879+0000","last_undegraded":"2024-09-18T12:00:58.344879+0000","last_fullsized":"2024-09-18T12:00:58.344879+0000","mapping_epoch":29,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23351296,"num_objects":5701,"num_object_clones":0,"num_object_copies":11402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10218,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13482","reported_seq":20612,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:54.202803+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:54.202803+0000","last_peered":"2024-09-18T12:00:54.202803+0000","last_clean":"2024-09-18T12:00:54.202803+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:54.202803+0000","last_undegraded":"2024-09-18T12:00:54.202803+0000","last_fullsized":"2024-09-18T12:00:54.202803+0000","mapping_epoch":38,"log_start":"33'3400","ondisk_log_start":"33'3400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24261842,"num_objects":5925,"num_object_clones":0,"num_object_copies":11850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13480,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13652","reported_seq":13732,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.252678+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:58.252678+0000","last_peered":"2024-09-18T12:00:58.252678+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:58.252678+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23733458,"num_objects":5796,"num_object_clones":0,"num_object_copies":11592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13650,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2264}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13720","reported_seq":13778,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.385841+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:58.385841+0000","last_peered":"2024-09-18T12:00:58.385841+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:58.385841+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23524562,"num_objects":5745,"num_object_clones":0,"num_object_copies":11490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13718,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13776","reported_seq":18366,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.466090+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:58.466090+0000","last_peered":"2024-09-18T12:00:58.466090+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:58.466090+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23437312,"num_objects":5722,"num_object_clones":0,"num_object_copies":11444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2189,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13757","reported_seq":13840,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.390415+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:58.390415+0000","last_peered":"2024-09-18T12:00:58.390415+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:58.390415+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23638633,"num_objects":5772,"num_object_clones":0,"num_object_copies":11544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13756,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13634","reported_seq":20782,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:54.197592+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:54.197592+0000","last_peered":"2024-09-18T12:00:54.197592+0000","last_clean":"2024-09-18T12:00:54.197592+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:54.197592+0000","last_undegraded":"2024-09-18T12:00:54.197592+0000","last_fullsized":"2024-09-18T12:00:54.197592+0000","mapping_epoch":40,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24119099,"num_objects":5891,"num_object_clones":0,"num_object_copies":11782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13631,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13614","reported_seq":13697,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.340785+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:58.340785+0000","last_peered":"2024-09-18T12:00:58.340785+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:58.340785+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22873298,"num_objects":5586,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13612,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13725","reported_seq":13812,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.315852+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:58.315852+0000","last_peered":"2024-09-18T12:00:58.315852+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:58.315852+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23706020,"num_objects":5791,"num_object_clones":0,"num_object_copies":11582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13722,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2268}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":377409612,"num_objects":92159,"num_object_clones":0,"num_object_copies":184318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203056,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9217,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":932843520,"data_stored":932772754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160763,"ondisk_log_size":160763,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051669,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595816,"kb_used_data":297952,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93776024,"statfs":{"total":96636764160,"available":96026648576,"internally_reserved":0,"allocated":305102848,"data_stored":304864732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":7.1740000000000004},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.36599999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":7.2539999999999996}]}]},{"osd":1,"up_from":35,"seq":150323855431,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592460,"kb_used_data":282112,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93779380,"statfs":{"total":96636764160,"available":96030085120,"internally_reserved":0,"allocated":288882688,"data_stored":288588765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.186},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.0699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.215}]}]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635620,"kb_used_data":333660,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93736220,"statfs":{"total":96636764160,"available":95985889280,"internally_reserved":0,"allocated":341667840,"data_stored":341430174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339927040,"data_stored":339923561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":288333824,"data_stored":288274088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304582656,"data_stored":304575105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:04.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:04.786 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:04.818 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:04.836 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-18T12:01:03.189808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376361055,"num_objects":91905,"num_object_clones":0,"num_object_copies":183810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91905,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203314,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9218,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1809000,"kb_used_data":907140,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281306520,"statfs":{"total":289910292480,"available":288057876480,"internally_reserved":0,"allocated":928911360,"data_stored":928141655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,4,7,9,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17739776,"num_objects":-4331,"num_object_clones":0,"num_object_copies":-8662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005360"},"pg_stats":[{"pgid":"2.f","version":"41'13788","reported_seq":13860,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.364957+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:00:58.364957+0000","last_peered":"2024-09-18T12:00:58.364957+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:00:58.364957+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23543808,"num_objects":5748,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13788,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2149}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13727","reported_seq":13887,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:58.425458+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:00:58.425458+0000","last_peered":"2024-09-18T12:00:58.425458+0000","last_clean":"2024-09-18T12:00:58.425458+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:00:58.425458+0000","last_undegraded":"2024-09-18T12:00:58.425458+0000","last_fullsized":"2024-09-18T12:00:58.425458+0000","mapping_epoch":29,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23367680,"num_objects":5705,"num_object_clones":0,"num_object_copies":11410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10919,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13745","reported_seq":13933,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.338329+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:00:58.338329+0000","last_peered":"2024-09-18T12:00:58.338329+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:00:58.338329+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23805976,"num_objects":5813,"num_object_clones":0,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10603,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2277}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13550","reported_seq":13581,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:58.377357+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:00:58.377357+0000","last_peered":"2024-09-18T12:00:58.377357+0000","last_clean":"2024-09-18T12:00:58.377357+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:00:58.377357+0000","last_undegraded":"2024-09-18T12:00:58.377357+0000","last_fullsized":"2024-09-18T12:00:58.377357+0000","mapping_epoch":29,"log_start":"34'3500","ondisk_log_start":"34'3500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23388160,"num_objects":5710,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13550,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13669","reported_seq":13862,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:58.276327+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:00:58.276327+0000","last_peered":"2024-09-18T12:00:58.276327+0000","last_clean":"2024-09-18T12:00:58.276327+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:00:58.276327+0000","last_undegraded":"2024-09-18T12:00:58.276327+0000","last_fullsized":"2024-09-18T12:00:58.276327+0000","mapping_epoch":29,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23666688,"num_objects":5778,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10834,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13661","reported_seq":13744,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.307629+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:00:58.307629+0000","last_peered":"2024-09-18T12:00:58.307629+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:00:58.307629+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23602386,"num_objects":5764,"num_object_clones":0,"num_object_copies":11528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13659,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13662","reported_seq":13751,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.384882+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:00:58.384882+0000","last_peered":"2024-09-18T12:00:58.384882+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:00:58.384882+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23389394,"num_objects":5712,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13660,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'13901","reported_seq":14128,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:58.344879+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:00:58.344879+0000","last_peered":"2024-09-18T12:00:58.344879+0000","last_clean":"2024-09-18T12:00:58.344879+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:00:58.344879+0000","last_undegraded":"2024-09-18T12:00:58.344879+0000","last_fullsized":"2024-09-18T12:00:58.344879+0000","mapping_epoch":29,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23351296,"num_objects":5701,"num_object_clones":0,"num_object_copies":11402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10218,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13618","reported_seq":20748,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:59.155341+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:00:59.155341+0000","last_peered":"2024-09-18T12:00:59.155341+0000","last_clean":"2024-09-18T12:00:59.155341+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:00:59.155341+0000","last_undegraded":"2024-09-18T12:00:59.155341+0000","last_fullsized":"2024-09-18T12:00:59.155341+0000","mapping_epoch":38,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23704786,"num_objects":5789,"num_object_clones":0,"num_object_copies":11578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13616,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13652","reported_seq":13732,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.252678+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:00:58.252678+0000","last_peered":"2024-09-18T12:00:58.252678+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:00:58.252678+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23733458,"num_objects":5796,"num_object_clones":0,"num_object_copies":11592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13650,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2264}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13720","reported_seq":13778,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.385841+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:00:58.385841+0000","last_peered":"2024-09-18T12:00:58.385841+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:00:58.385841+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23524562,"num_objects":5745,"num_object_clones":0,"num_object_copies":11490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13718,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13776","reported_seq":18366,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.466090+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:00:58.466090+0000","last_peered":"2024-09-18T12:00:58.466090+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:00:58.466090+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23437312,"num_objects":5722,"num_object_clones":0,"num_object_copies":11444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10256,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2189,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13757","reported_seq":13840,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.390415+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:00:58.390415+0000","last_peered":"2024-09-18T12:00:58.390415+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:00:58.390415+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23638633,"num_objects":5772,"num_object_clones":0,"num_object_copies":11544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13756,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13754","reported_seq":20902,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:00:59.195678+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:00:59.195678+0000","last_peered":"2024-09-18T12:00:59.195678+0000","last_clean":"2024-09-18T12:00:59.195678+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:00:59.195678+0000","last_undegraded":"2024-09-18T12:00:59.195678+0000","last_fullsized":"2024-09-18T12:00:59.195678+0000","mapping_epoch":40,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23627579,"num_objects":5771,"num_object_clones":0,"num_object_copies":11542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13751,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13614","reported_seq":13697,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.340785+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:00:58.340785+0000","last_peered":"2024-09-18T12:00:58.340785+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:00:58.340785+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'3600","ondisk_log_start":"37'3600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22873298,"num_objects":5586,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13612,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1993}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13725","reported_seq":13812,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:00:58.315852+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:00:58.315852+0000","last_peered":"2024-09-18T12:00:58.315852+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:00:58.315852+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23706020,"num_objects":5791,"num_object_clones":0,"num_object_copies":11582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13722,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2268}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":376361036,"num_objects":91903,"num_object_clones":0,"num_object_copies":183806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91903,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203312,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9217,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":926208000,"data_stored":926137234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160719,"ondisk_log_size":160719,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051669,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595816,"kb_used_data":297952,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93776024,"statfs":{"total":96636764160,"available":96026648576,"internally_reserved":0,"allocated":305102848,"data_stored":304864732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":7.1740000000000004},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.36599999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":7.2539999999999996}]}]},{"osd":1,"up_from":35,"seq":150323855432,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577564,"kb_used_data":275528,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93794276,"statfs":{"total":96636764160,"available":96045338624,"internally_reserved":0,"allocated":282140672,"data_stored":281846749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.19700000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.24199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.311}]}]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635620,"kb_used_data":333660,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93736220,"statfs":{"total":96636764160,"available":95985889280,"internally_reserved":0,"allocated":341667840,"data_stored":341430174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339927040,"data_stored":339923561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":281698304,"data_stored":281638568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304582656,"data_stored":304575105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:04.839 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:05.393 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:05.409 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:05.437 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-09-18T12:01:05.190231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":367673439,"num_objects":89784,"num_object_clones":0,"num_object_copies":179568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89784,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":205435,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9369,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160943,"ondisk_log_size":160943,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1807388,"kb_used_data":888884,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281308132,"statfs":{"total":289910292480,"available":288059527168,"internally_reserved":0,"allocated":910217216,"data_stored":909447511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,6,7,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26427392,"num_objects":-6452,"num_object_clones":0,"num_object_copies":-12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6452,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002722"},"pg_stats":[{"pgid":"2.f","version":"41'13957","reported_seq":14030,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.324841+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:03.324841+0000","last_peered":"2024-09-18T12:01:03.324841+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:03.324841+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22851584,"num_objects":5579,"num_object_clones":0,"num_object_copies":11158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13957,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1980}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13861","reported_seq":14020,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:03.423570+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:03.423570+0000","last_peered":"2024-09-18T12:01:03.423570+0000","last_clean":"2024-09-18T12:01:03.423570+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:03.423570+0000","last_undegraded":"2024-09-18T12:01:03.423570+0000","last_fullsized":"2024-09-18T12:01:03.423570+0000","mapping_epoch":29,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22818816,"num_objects":5571,"num_object_clones":0,"num_object_copies":11142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11053,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13877","reported_seq":14063,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.471075+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:03.471075+0000","last_peered":"2024-09-18T12:01:03.471075+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:03.471075+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265304,"num_objects":5681,"num_object_clones":0,"num_object_copies":11362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10735,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2145}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13701","reported_seq":13732,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:03.420328+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:03.420328+0000","last_peered":"2024-09-18T12:01:03.420328+0000","last_clean":"2024-09-18T12:01:03.420328+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:03.420328+0000","last_undegraded":"2024-09-18T12:01:03.420328+0000","last_fullsized":"2024-09-18T12:01:03.420328+0000","mapping_epoch":29,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22769664,"num_objects":5559,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13701,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13785","reported_seq":13978,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:03.315142+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:03.315142+0000","last_peered":"2024-09-18T12:01:03.315142+0000","last_clean":"2024-09-18T12:01:03.315142+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:03.315142+0000","last_undegraded":"2024-09-18T12:01:03.315142+0000","last_fullsized":"2024-09-18T12:01:03.315142+0000","mapping_epoch":29,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23191552,"num_objects":5662,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13785","reported_seq":13868,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.380878+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:03.380878+0000","last_peered":"2024-09-18T12:01:03.380878+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:03.380878+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23094482,"num_objects":5640,"num_object_clones":0,"num_object_copies":11280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13783,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13794","reported_seq":13883,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.344567+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:03.344567+0000","last_peered":"2024-09-18T12:01:03.344567+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:03.344567+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22848722,"num_objects":5580,"num_object_clones":0,"num_object_copies":11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13792,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2021}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14005","reported_seq":14232,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:03.386584+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:03.386584+0000","last_peered":"2024-09-18T12:01:03.386584+0000","last_clean":"2024-09-18T12:01:03.386584+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:03.386584+0000","last_undegraded":"2024-09-18T12:01:03.386584+0000","last_fullsized":"2024-09-18T12:01:03.386584+0000","mapping_epoch":29,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22925312,"num_objects":5597,"num_object_clones":0,"num_object_copies":11194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10322,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13756","reported_seq":20886,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:04.089000+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:04.089000+0000","last_peered":"2024-09-18T12:01:04.089000+0000","last_clean":"2024-09-18T12:01:04.089000+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:04.089000+0000","last_undegraded":"2024-09-18T12:01:04.089000+0000","last_fullsized":"2024-09-18T12:01:04.089000+0000","mapping_epoch":38,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23139538,"num_objects":5651,"num_object_clones":0,"num_object_copies":11302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13754,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13771","reported_seq":13851,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.329857+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:03.329857+0000","last_peered":"2024-09-18T12:01:03.329857+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:03.329857+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23246034,"num_objects":5677,"num_object_clones":0,"num_object_copies":11354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13769,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2145}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13857","reported_seq":13915,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.338108+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:03.338108+0000","last_peered":"2024-09-18T12:01:03.338108+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:03.338108+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22963410,"num_objects":5608,"num_object_clones":0,"num_object_copies":11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13855,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2058}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13909","reported_seq":18801,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.420937+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:03.420937+0000","last_peered":"2024-09-18T12:01:03.420937+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:03.420937+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22892544,"num_objects":5589,"num_object_clones":0,"num_object_copies":11178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10389,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2340,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4334}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13892","reported_seq":13976,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.323731+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:03.323731+0000","last_peered":"2024-09-18T12:01:03.323731+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:03.323731+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23085673,"num_objects":5637,"num_object_clones":0,"num_object_copies":11274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13891,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2102}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13889","reported_seq":21037,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:04.140506+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:04.140506+0000","last_peered":"2024-09-18T12:01:04.140506+0000","last_clean":"2024-09-18T12:01:04.140506+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:04.140506+0000","last_undegraded":"2024-09-18T12:01:04.140506+0000","last_fullsized":"2024-09-18T12:01:04.140506+0000","mapping_epoch":40,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23074619,"num_objects":5636,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13886,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13728","reported_seq":13809,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.387993+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:03.387993+0000","last_peered":"2024-09-18T12:01:03.387993+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:03.387993+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22406354,"num_objects":5472,"num_object_clones":0,"num_object_copies":10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13726,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13873","reported_seq":13960,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.336821+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:03.336821+0000","last_peered":"2024-09-18T12:01:03.336821+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:03.336821+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23099812,"num_objects":5643,"num_object_clones":0,"num_object_copies":11286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13870,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2120}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":367673420,"num_objects":89782,"num_object_clones":0,"num_object_copies":179564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89782,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":205433,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9368,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":908820480,"data_stored":908749714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160940,"ondisk_log_size":160940,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051670,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595816,"kb_used_data":293780,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93776024,"statfs":{"total":96636764160,"available":96026648576,"internally_reserved":0,"allocated":300830720,"data_stored":300592604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.5},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.873}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.52900000000000003}]}]},{"osd":1,"up_from":35,"seq":150323855433,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575192,"kb_used_data":269004,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93796648,"statfs":{"total":96636764160,"available":96047767552,"internally_reserved":0,"allocated":275460096,"data_stored":275166173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.219},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.26200000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.0179999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.34499999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636380,"kb_used_data":326100,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93735460,"statfs":{"total":96636764160,"available":95985111040,"internally_reserved":0,"allocated":333926400,"data_stored":333688734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333484032,"data_stored":333480553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275095552,"data_stored":275035816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":300240896,"data_stored":300233345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:05.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:01:05.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'13957", 'reported_seq': 14030, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:03.324841+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:01:03.324841+0000', 'last_peered': '2024-09-18T12:01:03.324841+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:01:03.324841+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'3900", 'ondisk_log_start': "37'3900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22851584, 'num_objects': 5579, 'num_object_clones': 0, 'num_object_copies': 11158, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13957, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1980}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:05.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:01:05.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'13877", 'reported_seq': 14063, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:03.471075+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:01:03.471075+0000', 'last_peered': '2024-09-18T12:01:03.471075+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:01:03.471075+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'3800", 'ondisk_log_start': "37'3800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23265304, 'num_objects': 5681, 'num_object_clones': 0, 'num_object_copies': 11362, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5681, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10735, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2145}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:05.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:01:05.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'13785", 'reported_seq': 13868, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:03.380878+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:01:03.380878+0000', 'last_peered': '2024-09-18T12:01:03.380878+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:01:03.380878+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'3700", 'ondisk_log_start': "37'3700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23094482, 'num_objects': 5640, 'num_object_clones': 0, 'num_object_copies': 11280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5640, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13783, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2113}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:05.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:01:05.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'13794", 'reported_seq': 13883, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:03.344567+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:01:03.344567+0000', 'last_peered': '2024-09-18T12:01:03.344567+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:01:03.344567+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'3700", 'ondisk_log_start': "37'3700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22848722, 'num_objects': 5580, 'num_object_clones': 0, 'num_object_copies': 11160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13792, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2021}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:05.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:01:05.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'13771", 'reported_seq': 13851, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:03.329857+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:01:03.329857+0000', 'last_peered': '2024-09-18T12:01:03.329857+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:01:03.329857+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'3700", 'ondisk_log_start': "37'3700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23246034, 'num_objects': 5677, 'num_object_clones': 0, 'num_object_copies': 11354, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13769, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2145}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:05.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:01:05.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'13857", 'reported_seq': 13915, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:03.338108+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:01:03.338108+0000', 'last_peered': '2024-09-18T12:01:03.338108+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:01:03.338108+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'3800", 'ondisk_log_start': "37'3800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22963410, 'num_objects': 5608, 'num_object_clones': 0, 'num_object_copies': 11216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13855, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2058}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:05.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:01:05.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'13909", 'reported_seq': 18801, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:03.420937+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:01:03.420937+0000', 'last_peered': '2024-09-18T12:01:03.420937+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:01:03.420937+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "37'3900", 'ondisk_log_start': "37'3900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22892544, 'num_objects': 5589, 'num_object_clones': 0, 'num_object_copies': 11178, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1255, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10389, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2340, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 4334}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:05.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:01:05.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'13892", 'reported_seq': 13976, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:03.323731+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:01:03.323731+0000', 'last_peered': '2024-09-18T12:01:03.323731+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:01:03.323731+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'3800", 'ondisk_log_start': "37'3800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23085673, 'num_objects': 5637, 'num_object_clones': 0, 'num_object_copies': 11274, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5637, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13891, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2102}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:05.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:01:05.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'13728", 'reported_seq': 13809, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:03.387993+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:01:03.387993+0000', 'last_peered': '2024-09-18T12:01:03.387993+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:01:03.387993+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'3700", 'ondisk_log_start': "37'3700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22406354, 'num_objects': 5472, 'num_object_clones': 0, 'num_object_copies': 10944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13726, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:05.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:01:05.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'13873", 'reported_seq': 13960, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:03.336821+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:01:03.336821+0000', 'last_peered': '2024-09-18T12:01:03.336821+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:01:03.336821+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'3800", 'ondisk_log_start': "37'3800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23099812, 'num_objects': 5643, 'num_object_clones': 0, 'num_object_copies': 11286, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5643, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13870, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2120}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:05.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:01:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:06.441 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33224/179568 objects degraded (18.502%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":466,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":89784,"data_bytes":367673439,"bytes_used":1850765312,"bytes_avail":288059527168,"bytes_total":289910292480,"degraded_objects":33224,"degraded_total":179568,"degraded_ratio":0.18502183017018622,"recovering_objects_per_sec":35,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":421,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":537},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:01:06.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:01:06.494 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:07.282 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:07.313 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:07.339 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-18T12:01:07.190703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":367673439,"num_objects":89784,"num_object_clones":0,"num_object_copies":179568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89784,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":205435,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9369,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160943,"ondisk_log_size":160943,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1807388,"kb_used_data":888884,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281308132,"statfs":{"total":289910292480,"available":288059527168,"internally_reserved":0,"allocated":910217216,"data_stored":909447511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,6,7,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17616896,"num_objects":-4301,"num_object_clones":0,"num_object_copies":-8602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4301,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002750"},"pg_stats":[{"pgid":"2.f","version":"41'13957","reported_seq":14030,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.324841+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:03.324841+0000","last_peered":"2024-09-18T12:01:03.324841+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:03.324841+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22851584,"num_objects":5579,"num_object_clones":0,"num_object_copies":11158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13957,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1980}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'13861","reported_seq":14020,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:03.423570+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:03.423570+0000","last_peered":"2024-09-18T12:01:03.423570+0000","last_clean":"2024-09-18T12:01:03.423570+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:03.423570+0000","last_undegraded":"2024-09-18T12:01:03.423570+0000","last_fullsized":"2024-09-18T12:01:03.423570+0000","mapping_epoch":29,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22818816,"num_objects":5571,"num_object_clones":0,"num_object_copies":11142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11053,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'13877","reported_seq":14063,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.471075+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:03.471075+0000","last_peered":"2024-09-18T12:01:03.471075+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:03.471075+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265304,"num_objects":5681,"num_object_clones":0,"num_object_copies":11362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10735,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2145}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13701","reported_seq":13732,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:03.420328+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:03.420328+0000","last_peered":"2024-09-18T12:01:03.420328+0000","last_clean":"2024-09-18T12:01:03.420328+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:03.420328+0000","last_undegraded":"2024-09-18T12:01:03.420328+0000","last_fullsized":"2024-09-18T12:01:03.420328+0000","mapping_epoch":29,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22769664,"num_objects":5559,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13701,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13785","reported_seq":13978,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:03.315142+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:03.315142+0000","last_peered":"2024-09-18T12:01:03.315142+0000","last_clean":"2024-09-18T12:01:03.315142+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:03.315142+0000","last_undegraded":"2024-09-18T12:01:03.315142+0000","last_fullsized":"2024-09-18T12:01:03.315142+0000","mapping_epoch":29,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23191552,"num_objects":5662,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13785","reported_seq":13868,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.380878+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:03.380878+0000","last_peered":"2024-09-18T12:01:03.380878+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:03.380878+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23094482,"num_objects":5640,"num_object_clones":0,"num_object_copies":11280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13783,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13794","reported_seq":13883,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.344567+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:03.344567+0000","last_peered":"2024-09-18T12:01:03.344567+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:03.344567+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22848722,"num_objects":5580,"num_object_clones":0,"num_object_copies":11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13792,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2021}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14005","reported_seq":14232,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:03.386584+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:03.386584+0000","last_peered":"2024-09-18T12:01:03.386584+0000","last_clean":"2024-09-18T12:01:03.386584+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:03.386584+0000","last_undegraded":"2024-09-18T12:01:03.386584+0000","last_fullsized":"2024-09-18T12:01:03.386584+0000","mapping_epoch":29,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22925312,"num_objects":5597,"num_object_clones":0,"num_object_copies":11194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10322,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13756","reported_seq":20886,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:04.089000+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:04.089000+0000","last_peered":"2024-09-18T12:01:04.089000+0000","last_clean":"2024-09-18T12:01:04.089000+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:04.089000+0000","last_undegraded":"2024-09-18T12:01:04.089000+0000","last_fullsized":"2024-09-18T12:01:04.089000+0000","mapping_epoch":38,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23139538,"num_objects":5651,"num_object_clones":0,"num_object_copies":11302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13754,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13771","reported_seq":13851,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.329857+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:03.329857+0000","last_peered":"2024-09-18T12:01:03.329857+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:03.329857+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23246034,"num_objects":5677,"num_object_clones":0,"num_object_copies":11354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13769,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2145}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13857","reported_seq":13915,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.338108+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:03.338108+0000","last_peered":"2024-09-18T12:01:03.338108+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:03.338108+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22963410,"num_objects":5608,"num_object_clones":0,"num_object_copies":11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13855,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2058}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'13909","reported_seq":18801,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.420937+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:03.420937+0000","last_peered":"2024-09-18T12:01:03.420937+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:03.420937+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22892544,"num_objects":5589,"num_object_clones":0,"num_object_copies":11178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10389,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2340,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4334}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'13892","reported_seq":13976,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.323731+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:03.323731+0000","last_peered":"2024-09-18T12:01:03.323731+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:03.323731+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23085673,"num_objects":5637,"num_object_clones":0,"num_object_copies":11274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13891,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2102}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13889","reported_seq":21037,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:04.140506+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:04.140506+0000","last_peered":"2024-09-18T12:01:04.140506+0000","last_clean":"2024-09-18T12:01:04.140506+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:04.140506+0000","last_undegraded":"2024-09-18T12:01:04.140506+0000","last_fullsized":"2024-09-18T12:01:04.140506+0000","mapping_epoch":40,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23074619,"num_objects":5636,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13886,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13728","reported_seq":13809,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.387993+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:03.387993+0000","last_peered":"2024-09-18T12:01:03.387993+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:03.387993+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22406354,"num_objects":5472,"num_object_clones":0,"num_object_copies":10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13726,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'13873","reported_seq":13960,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:03.336821+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:03.336821+0000","last_peered":"2024-09-18T12:01:03.336821+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:03.336821+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23099812,"num_objects":5643,"num_object_clones":0,"num_object_copies":11286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13870,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2120}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":367673420,"num_objects":89782,"num_object_clones":0,"num_object_copies":179564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89782,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":205433,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9368,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":908820480,"data_stored":908749714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160940,"ondisk_log_size":160940,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051670,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595816,"kb_used_data":293780,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93776024,"statfs":{"total":96636764160,"available":96026648576,"internally_reserved":0,"allocated":300830720,"data_stored":300592604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.5},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.873}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.52900000000000003}]}]},{"osd":1,"up_from":35,"seq":150323855433,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575192,"kb_used_data":269004,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93796648,"statfs":{"total":96636764160,"available":96047767552,"internally_reserved":0,"allocated":275460096,"data_stored":275166173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.219},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.26200000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.0179999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.34499999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636380,"kb_used_data":326100,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93735460,"statfs":{"total":96636764160,"available":95985111040,"internally_reserved":0,"allocated":333926400,"data_stored":333688734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333484032,"data_stored":333480553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275095552,"data_stored":275035816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":300240896,"data_stored":300233345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:10.341 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:11.090 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:11.111 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:11.143 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-09-18T12:01:09.191140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":360161375,"num_objects":87950,"num_object_clones":0,"num_object_copies":175900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87950,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207269,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9514,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160577,"ondisk_log_size":160577,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1788516,"kb_used_data":878344,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281327004,"statfs":{"total":289910292480,"available":288078852096,"internally_reserved":0,"allocated":899424256,"data_stored":898654551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,5,5,5,5,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25128960,"num_objects":-6135,"num_object_clones":0,"num_object_copies":-12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6135,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":428,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002666"},"pg_stats":[{"pgid":"2.f","version":"41'14090","reported_seq":14163,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.393392+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:08.393392+0000","last_peered":"2024-09-18T12:01:08.393392+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:08.393392+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22306816,"num_objects":5446,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14090,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1847}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14003","reported_seq":14162,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:08.483535+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:08.483535+0000","last_peered":"2024-09-18T12:01:08.483535+0000","last_clean":"2024-09-18T12:01:08.483535+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:08.483535+0000","last_undegraded":"2024-09-18T12:01:08.483535+0000","last_fullsized":"2024-09-18T12:01:08.483535+0000","mapping_epoch":29,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22237184,"num_objects":5429,"num_object_clones":0,"num_object_copies":10858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14016","reported_seq":14204,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.403093+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:08.403093+0000","last_peered":"2024-09-18T12:01:08.403093+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:08.403093+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22695960,"num_objects":5542,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10874,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2006}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13833","reported_seq":13864,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:08.408586+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:08.408586+0000","last_peered":"2024-09-18T12:01:08.408586+0000","last_clean":"2024-09-18T12:01:08.408586+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:08.408586+0000","last_undegraded":"2024-09-18T12:01:08.408586+0000","last_fullsized":"2024-09-18T12:01:08.408586+0000","mapping_epoch":29,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22228992,"num_objects":5427,"num_object_clones":0,"num_object_copies":10854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13833,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13913","reported_seq":14106,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:08.391114+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:08.391114+0000","last_peered":"2024-09-18T12:01:08.391114+0000","last_clean":"2024-09-18T12:01:08.391114+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:08.391114+0000","last_undegraded":"2024-09-18T12:01:08.391114+0000","last_fullsized":"2024-09-18T12:01:08.391114+0000","mapping_epoch":29,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22667264,"num_objects":5534,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11078,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13907","reported_seq":13989,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.394275+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:08.394275+0000","last_peered":"2024-09-18T12:01:08.394275+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:08.394275+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22594770,"num_objects":5518,"num_object_clones":0,"num_object_copies":11036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13905,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1991}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13917","reported_seq":14005,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.409796+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:08.409796+0000","last_peered":"2024-09-18T12:01:08.409796+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:08.409796+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22344914,"num_objects":5457,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13915,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14131","reported_seq":14358,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:08.318705+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:08.318705+0000","last_peered":"2024-09-18T12:01:08.318705+0000","last_clean":"2024-09-18T12:01:08.318705+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:08.318705+0000","last_undegraded":"2024-09-18T12:01:08.318705+0000","last_fullsized":"2024-09-18T12:01:08.318705+0000","mapping_epoch":29,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22409216,"num_objects":5471,"num_object_clones":0,"num_object_copies":10942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13756","reported_seq":20886,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:04.089000+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:04.089000+0000","last_peered":"2024-09-18T12:01:04.089000+0000","last_clean":"2024-09-18T12:01:04.089000+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:04.089000+0000","last_undegraded":"2024-09-18T12:01:04.089000+0000","last_fullsized":"2024-09-18T12:01:04.089000+0000","mapping_epoch":38,"log_start":"37'3700","ondisk_log_start":"37'3700","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23139538,"num_objects":5651,"num_object_clones":0,"num_object_copies":11302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13754,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13921","reported_seq":14001,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.324783+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:08.324783+0000","last_peered":"2024-09-18T12:01:08.324783+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:08.324783+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22631634,"num_objects":5527,"num_object_clones":0,"num_object_copies":11054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13919,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1995}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13991","reported_seq":14049,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.463710+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:08.463710+0000","last_peered":"2024-09-18T12:01:08.463710+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:08.463710+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22414546,"num_objects":5474,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13989,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1924}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14032","reported_seq":19214,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.435466+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:08.435466+0000","last_peered":"2024-09-18T12:01:08.435466+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:08.435466+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22388736,"num_objects":5466,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10512,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2485,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4356}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14014","reported_seq":14098,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.394402+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:08.394402+0000","last_peered":"2024-09-18T12:01:08.394402+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:08.394402+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22585961,"num_objects":5515,"num_object_clones":0,"num_object_copies":11030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14013,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'13889","reported_seq":21037,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:04.140506+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:04.140506+0000","last_peered":"2024-09-18T12:01:04.140506+0000","last_clean":"2024-09-18T12:01:04.140506+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:04.140506+0000","last_undegraded":"2024-09-18T12:01:04.140506+0000","last_fullsized":"2024-09-18T12:01:04.140506+0000","mapping_epoch":40,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23074619,"num_objects":5636,"num_object_clones":0,"num_object_copies":11272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13886,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13856","reported_seq":13939,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.394451+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:08.394451+0000","last_peered":"2024-09-18T12:01:08.394451+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:08.394451+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21882066,"num_objects":5344,"num_object_clones":0,"num_object_copies":10688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13854,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14005","reported_seq":14092,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.441826+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:08.441826+0000","last_peered":"2024-09-18T12:01:08.441826+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:08.441826+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22559140,"num_objects":5511,"num_object_clones":0,"num_object_copies":11022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14002,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1988}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":360161356,"num_objects":87948,"num_object_clones":0,"num_object_copies":175896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207267,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9513,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":898113536,"data_stored":898042770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160574,"ondisk_log_size":160574,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051671,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595772,"kb_used_data":289588,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93776068,"statfs":{"total":96636764160,"available":96026693632,"internally_reserved":0,"allocated":296538112,"data_stored":296299996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.249},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.33200000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.378},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.29499999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855433,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575192,"kb_used_data":269004,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93796648,"statfs":{"total":96636764160,"available":96047767552,"internally_reserved":0,"allocated":275460096,"data_stored":275166173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.219},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.26200000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":3.0179999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.34499999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617552,"kb_used_data":319752,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93754288,"statfs":{"total":96636764160,"available":96004390912,"internally_reserved":0,"allocated":327426048,"data_stored":327188382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.7409999999999997},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.35799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":6.6120000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327045120,"data_stored":327041641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275095552,"data_stored":275035816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295972864,"data_stored":295965313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:11.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:11.925 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:11.959 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:11.967 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-18T12:01:11.191625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":359063647,"num_objects":87682,"num_object_clones":0,"num_object_copies":175364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87682,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207537,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9514,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160545,"ondisk_log_size":160545,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1773908,"kb_used_data":872060,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281341612,"statfs":{"total":289910292480,"available":288093810688,"internally_reserved":0,"allocated":892989440,"data_stored":892219735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,5,4,4,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18345984,"num_objects":-4479,"num_object_clones":0,"num_object_copies":-8958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4479,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002727"},"pg_stats":[{"pgid":"2.f","version":"41'14090","reported_seq":14163,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.393392+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:08.393392+0000","last_peered":"2024-09-18T12:01:08.393392+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:08.393392+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22306816,"num_objects":5446,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14090,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1847}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14003","reported_seq":14162,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:08.483535+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:08.483535+0000","last_peered":"2024-09-18T12:01:08.483535+0000","last_clean":"2024-09-18T12:01:08.483535+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:08.483535+0000","last_undegraded":"2024-09-18T12:01:08.483535+0000","last_fullsized":"2024-09-18T12:01:08.483535+0000","mapping_epoch":29,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22237184,"num_objects":5429,"num_object_clones":0,"num_object_copies":10858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14016","reported_seq":14204,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.403093+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:08.403093+0000","last_peered":"2024-09-18T12:01:08.403093+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:08.403093+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22695960,"num_objects":5542,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10874,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2006}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13833","reported_seq":13864,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:08.408586+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:08.408586+0000","last_peered":"2024-09-18T12:01:08.408586+0000","last_clean":"2024-09-18T12:01:08.408586+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:08.408586+0000","last_undegraded":"2024-09-18T12:01:08.408586+0000","last_fullsized":"2024-09-18T12:01:08.408586+0000","mapping_epoch":29,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22228992,"num_objects":5427,"num_object_clones":0,"num_object_copies":10854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13833,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13913","reported_seq":14106,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:08.391114+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:08.391114+0000","last_peered":"2024-09-18T12:01:08.391114+0000","last_clean":"2024-09-18T12:01:08.391114+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:08.391114+0000","last_undegraded":"2024-09-18T12:01:08.391114+0000","last_fullsized":"2024-09-18T12:01:08.391114+0000","mapping_epoch":29,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22667264,"num_objects":5534,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11078,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13907","reported_seq":13989,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.394275+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:08.394275+0000","last_peered":"2024-09-18T12:01:08.394275+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:08.394275+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22594770,"num_objects":5518,"num_object_clones":0,"num_object_copies":11036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13905,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1991}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13917","reported_seq":14005,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.409796+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:08.409796+0000","last_peered":"2024-09-18T12:01:08.409796+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:08.409796+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22344914,"num_objects":5457,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13915,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14131","reported_seq":14358,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:08.318705+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:08.318705+0000","last_peered":"2024-09-18T12:01:08.318705+0000","last_clean":"2024-09-18T12:01:08.318705+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:08.318705+0000","last_undegraded":"2024-09-18T12:01:08.318705+0000","last_fullsized":"2024-09-18T12:01:08.318705+0000","mapping_epoch":29,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22409216,"num_objects":5471,"num_object_clones":0,"num_object_copies":10942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13897","reported_seq":21027,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:09.186998+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:09.186998+0000","last_peered":"2024-09-18T12:01:09.186998+0000","last_clean":"2024-09-18T12:01:09.186998+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:09.186998+0000","last_undegraded":"2024-09-18T12:01:09.186998+0000","last_fullsized":"2024-09-18T12:01:09.186998+0000","mapping_epoch":38,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22562002,"num_objects":5510,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13895,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13921","reported_seq":14001,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.324783+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:08.324783+0000","last_peered":"2024-09-18T12:01:08.324783+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:08.324783+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22631634,"num_objects":5527,"num_object_clones":0,"num_object_copies":11054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13919,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1995}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13991","reported_seq":14049,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.463710+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:08.463710+0000","last_peered":"2024-09-18T12:01:08.463710+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:08.463710+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22414546,"num_objects":5474,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13989,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1924}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14032","reported_seq":19214,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.435466+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:08.435466+0000","last_peered":"2024-09-18T12:01:08.435466+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:08.435466+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22388736,"num_objects":5466,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10512,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2485,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4356}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14014","reported_seq":14098,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.394402+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:08.394402+0000","last_peered":"2024-09-18T12:01:08.394402+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:08.394402+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22585961,"num_objects":5515,"num_object_clones":0,"num_object_copies":11030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14013,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14016","reported_seq":21164,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:09.274387+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:09.274387+0000","last_peered":"2024-09-18T12:01:09.274387+0000","last_clean":"2024-09-18T12:01:09.274387+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:09.274387+0000","last_undegraded":"2024-09-18T12:01:09.274387+0000","last_fullsized":"2024-09-18T12:01:09.274387+0000","mapping_epoch":40,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22554427,"num_objects":5509,"num_object_clones":0,"num_object_copies":11018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14013,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13856","reported_seq":13939,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.394451+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:08.394451+0000","last_peered":"2024-09-18T12:01:08.394451+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:08.394451+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21882066,"num_objects":5344,"num_object_clones":0,"num_object_copies":10688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13854,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14005","reported_seq":14092,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.441826+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:08.441826+0000","last_peered":"2024-09-18T12:01:08.441826+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:08.441826+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22559140,"num_objects":5511,"num_object_clones":0,"num_object_copies":11022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14002,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1988}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":359063628,"num_objects":87680,"num_object_clones":0,"num_object_copies":175360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87680,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207535,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9513,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":891559936,"data_stored":891489170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160542,"ondisk_log_size":160542,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051671,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595772,"kb_used_data":289588,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93776068,"statfs":{"total":96636764160,"available":96026693632,"internally_reserved":0,"allocated":296538112,"data_stored":296299996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.249},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.33200000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.378},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.29499999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855434,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560584,"kb_used_data":262720,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93811256,"statfs":{"total":96636764160,"available":96062726144,"internally_reserved":0,"allocated":269025280,"data_stored":268731357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.312}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":1.673},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.29699999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617552,"kb_used_data":319752,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93754288,"statfs":{"total":96636764160,"available":96004390912,"internally_reserved":0,"allocated":327426048,"data_stored":327188382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.7409999999999997},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.35799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":6.6120000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327045120,"data_stored":327041641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268541952,"data_stored":268482216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295972864,"data_stored":295965313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:11.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:01:11.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'14090", 'reported_seq': 14163, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:08.393392+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:01:08.393392+0000', 'last_peered': '2024-09-18T12:01:08.393392+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:01:08.393392+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'4000", 'ondisk_log_start': "37'4000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22306816, 'num_objects': 5446, 'num_object_clones': 0, 'num_object_copies': 10892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5446, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14090, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1847}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:11.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:01:11.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'14016", 'reported_seq': 14204, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:08.403093+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:01:08.403093+0000', 'last_peered': '2024-09-18T12:01:08.403093+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:01:08.403093+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'4000", 'ondisk_log_start': "37'4000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22695960, 'num_objects': 5542, 'num_object_clones': 0, 'num_object_copies': 11084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5542, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10874, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 2006}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:11.971 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:01:11.971 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'13907", 'reported_seq': 13989, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:08.394275+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:01:08.394275+0000', 'last_peered': '2024-09-18T12:01:08.394275+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:01:08.394275+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'3900", 'ondisk_log_start': "37'3900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22594770, 'num_objects': 5518, 'num_object_clones': 0, 'num_object_copies': 11036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13905, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1991}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:11.971 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:01:11.971 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'13917", 'reported_seq': 14005, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:08.409796+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:01:08.409796+0000', 'last_peered': '2024-09-18T12:01:08.409796+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:01:08.409796+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'3900", 'ondisk_log_start': "37'3900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22344914, 'num_objects': 5457, 'num_object_clones': 0, 'num_object_copies': 10914, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13915, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1898}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:11.972 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:01:11.972 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'13921", 'reported_seq': 14001, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:08.324783+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:01:08.324783+0000', 'last_peered': '2024-09-18T12:01:08.324783+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:01:08.324783+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'3900", 'ondisk_log_start': "37'3900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22631634, 'num_objects': 5527, 'num_object_clones': 0, 'num_object_copies': 11054, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13919, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1995}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:11.972 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:01:11.972 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'13991", 'reported_seq': 14049, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:08.463710+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:01:08.463710+0000', 'last_peered': '2024-09-18T12:01:08.463710+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:01:08.463710+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'3900", 'ondisk_log_start': "37'3900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22414546, 'num_objects': 5474, 'num_object_clones': 0, 'num_object_copies': 10948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13989, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1924}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:11.972 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:01:11.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'14032", 'reported_seq': 19214, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:08.435466+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:01:08.435466+0000', 'last_peered': '2024-09-18T12:01:08.435466+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:01:08.435466+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "37'4000", 'ondisk_log_start': "37'4000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22388736, 'num_objects': 5466, 'num_object_clones': 0, 'num_object_copies': 10932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1110, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10512, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2485, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 4356}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:11.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:01:11.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'14014", 'reported_seq': 14098, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:08.394402+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:01:08.394402+0000', 'last_peered': '2024-09-18T12:01:08.394402+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:01:08.394402+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'4000", 'ondisk_log_start': "37'4000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22585961, 'num_objects': 5515, 'num_object_clones': 0, 'num_object_copies': 11030, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14013, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1980}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:11.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:01:11.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'13856", 'reported_seq': 13939, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:08.394451+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:01:08.394451+0000', 'last_peered': '2024-09-18T12:01:08.394451+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:01:08.394451+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'3800", 'ondisk_log_start': "37'3800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21882066, 'num_objects': 5344, 'num_object_clones': 0, 'num_object_copies': 10688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5344, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13854, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1751}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:11.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:01:11.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'14005", 'reported_seq': 14092, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:08.441826+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:01:08.441826+0000', 'last_peered': '2024-09-18T12:01:08.441826+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:01:08.441826+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'4000", 'ondisk_log_start': "37'4000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22559140, 'num_objects': 5511, 'num_object_clones': 0, 'num_object_copies': 11022, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14002, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1988}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:11.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:01:12.909 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:12.910 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33079/175364 objects degraded (18.863%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":472,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":87682,"data_bytes":359063647,"bytes_used":1816481792,"bytes_avail":288093810688,"bytes_total":289910292480,"degraded_objects":33079,"degraded_total":175364,"degraded_ratio":0.18863050569101983,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":296,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":373},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:01:12.966 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:01:12.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:13.481 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:13.509 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:13.531 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-18T12:01:13.192077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":359063647,"num_objects":87682,"num_object_clones":0,"num_object_copies":175364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87682,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207537,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9514,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160545,"ondisk_log_size":160545,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1773908,"kb_used_data":872060,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281341612,"statfs":{"total":289910292480,"available":288093810688,"internally_reserved":0,"allocated":892989440,"data_stored":892219735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,5,4,4,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17297408,"num_objects":-4223,"num_object_clones":0,"num_object_copies":-8446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4223,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002707"},"pg_stats":[{"pgid":"2.f","version":"41'14090","reported_seq":14163,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.393392+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:08.393392+0000","last_peered":"2024-09-18T12:01:08.393392+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:08.393392+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22306816,"num_objects":5446,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14090,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1847}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14003","reported_seq":14162,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:08.483535+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:08.483535+0000","last_peered":"2024-09-18T12:01:08.483535+0000","last_clean":"2024-09-18T12:01:08.483535+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:08.483535+0000","last_undegraded":"2024-09-18T12:01:08.483535+0000","last_fullsized":"2024-09-18T12:01:08.483535+0000","mapping_epoch":29,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22237184,"num_objects":5429,"num_object_clones":0,"num_object_copies":10858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14016","reported_seq":14204,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.403093+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:08.403093+0000","last_peered":"2024-09-18T12:01:08.403093+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:08.403093+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22695960,"num_objects":5542,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10874,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":2006}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13833","reported_seq":13864,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:08.408586+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:08.408586+0000","last_peered":"2024-09-18T12:01:08.408586+0000","last_clean":"2024-09-18T12:01:08.408586+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:08.408586+0000","last_undegraded":"2024-09-18T12:01:08.408586+0000","last_fullsized":"2024-09-18T12:01:08.408586+0000","mapping_epoch":29,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22228992,"num_objects":5427,"num_object_clones":0,"num_object_copies":10854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13833,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'13913","reported_seq":14106,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:08.391114+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:08.391114+0000","last_peered":"2024-09-18T12:01:08.391114+0000","last_clean":"2024-09-18T12:01:08.391114+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:08.391114+0000","last_undegraded":"2024-09-18T12:01:08.391114+0000","last_fullsized":"2024-09-18T12:01:08.391114+0000","mapping_epoch":29,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22667264,"num_objects":5534,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11078,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'13907","reported_seq":13989,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.394275+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:08.394275+0000","last_peered":"2024-09-18T12:01:08.394275+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:08.394275+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22594770,"num_objects":5518,"num_object_clones":0,"num_object_copies":11036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13905,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1991}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'13917","reported_seq":14005,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.409796+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:08.409796+0000","last_peered":"2024-09-18T12:01:08.409796+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:08.409796+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22344914,"num_objects":5457,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13915,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14131","reported_seq":14358,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:08.318705+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:08.318705+0000","last_peered":"2024-09-18T12:01:08.318705+0000","last_clean":"2024-09-18T12:01:08.318705+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:08.318705+0000","last_undegraded":"2024-09-18T12:01:08.318705+0000","last_fullsized":"2024-09-18T12:01:08.318705+0000","mapping_epoch":29,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22409216,"num_objects":5471,"num_object_clones":0,"num_object_copies":10942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10448,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'13897","reported_seq":21027,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:09.186998+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:09.186998+0000","last_peered":"2024-09-18T12:01:09.186998+0000","last_clean":"2024-09-18T12:01:09.186998+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:09.186998+0000","last_undegraded":"2024-09-18T12:01:09.186998+0000","last_fullsized":"2024-09-18T12:01:09.186998+0000","mapping_epoch":38,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22562002,"num_objects":5510,"num_object_clones":0,"num_object_copies":11020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13895,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'13921","reported_seq":14001,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.324783+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:08.324783+0000","last_peered":"2024-09-18T12:01:08.324783+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:08.324783+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22631634,"num_objects":5527,"num_object_clones":0,"num_object_copies":11054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13919,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1995}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'13991","reported_seq":14049,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.463710+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:08.463710+0000","last_peered":"2024-09-18T12:01:08.463710+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:08.463710+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22414546,"num_objects":5474,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13989,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1924}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14032","reported_seq":19214,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.435466+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:08.435466+0000","last_peered":"2024-09-18T12:01:08.435466+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:08.435466+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22388736,"num_objects":5466,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10512,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2485,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4356}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14014","reported_seq":14098,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.394402+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:08.394402+0000","last_peered":"2024-09-18T12:01:08.394402+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:08.394402+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22585961,"num_objects":5515,"num_object_clones":0,"num_object_copies":11030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14013,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14016","reported_seq":21164,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:09.274387+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:09.274387+0000","last_peered":"2024-09-18T12:01:09.274387+0000","last_clean":"2024-09-18T12:01:09.274387+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:09.274387+0000","last_undegraded":"2024-09-18T12:01:09.274387+0000","last_fullsized":"2024-09-18T12:01:09.274387+0000","mapping_epoch":40,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22554427,"num_objects":5509,"num_object_clones":0,"num_object_copies":11018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14013,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13856","reported_seq":13939,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.394451+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:08.394451+0000","last_peered":"2024-09-18T12:01:08.394451+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:08.394451+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'3800","ondisk_log_start":"37'3800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21882066,"num_objects":5344,"num_object_clones":0,"num_object_copies":10688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13854,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14005","reported_seq":14092,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:08.441826+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:08.441826+0000","last_peered":"2024-09-18T12:01:08.441826+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:08.441826+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22559140,"num_objects":5511,"num_object_clones":0,"num_object_copies":11022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14002,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1988}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":359063628,"num_objects":87680,"num_object_clones":0,"num_object_copies":175360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87680,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207535,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9513,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":891559936,"data_stored":891489170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160542,"ondisk_log_size":160542,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051671,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595772,"kb_used_data":289588,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93776068,"statfs":{"total":96636764160,"available":96026693632,"internally_reserved":0,"allocated":296538112,"data_stored":296299996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.249},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.33200000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:00:13 2024","interfaces":[{"interface":"back","average":{"1min":2.7170000000000001,"5min":1.6850000000000001,"15min":1.95},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.1750000000000007,"5min":8.1750000000000007,"15min":8.1750000000000007},"last":0.378},{"interface":"front","average":{"1min":1.302,"5min":1.145,"15min":0.96099999999999997},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":7.54,"5min":11.109,"15min":11.109},"last":0.29499999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855434,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560584,"kb_used_data":262720,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93811256,"statfs":{"total":96636764160,"available":96062726144,"internally_reserved":0,"allocated":269025280,"data_stored":268731357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.40899999999999997,"15min":0.45100000000000001},"min":{"1min":0.20000000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.51200000000000001,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5069999999999999,"15min":1.155},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.1950000000000003,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.312}]},{"osd":2,"last update":"Wed Sep 18 12:00:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.5409999999999999,"15min":1.1120000000000001},"min":{"1min":0.22500000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.8849999999999998,"5min":11.147,"15min":11.147},"last":1.673},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.85399999999999998,"15min":0.70199999999999996},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.7560000000000002,"5min":11.996,"15min":11.996},"last":0.29699999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617552,"kb_used_data":319752,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93754288,"statfs":{"total":96636764160,"available":96004390912,"internally_reserved":0,"allocated":327426048,"data_stored":327188382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.7409999999999997},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.35799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":6.6120000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327045120,"data_stored":327041641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268541952,"data_stored":268482216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295972864,"data_stored":295965313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:16.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:17.253 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:17.267 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:17.285 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-09-18T12:01:17.192961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":350163039,"num_objects":85509,"num_object_clones":0,"num_object_copies":171018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":209710,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9648,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160818,"ondisk_log_size":160818,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1773248,"kb_used_data":854760,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281342272,"statfs":{"total":289910292480,"available":288094486528,"internally_reserved":0,"allocated":875274240,"data_stored":874504535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,5,13,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17510400,"num_objects":-4275,"num_object_clones":0,"num_object_copies":-8550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4275,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002729"},"pg_stats":[{"pgid":"2.f","version":"41'14231","reported_seq":14302,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.450333+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:13.450333+0000","last_peered":"2024-09-18T12:01:13.450333+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:13.450333+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21729280,"num_objects":5305,"num_object_clones":0,"num_object_copies":10610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14231,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1706}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14129","reported_seq":14289,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:13.400146+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:13.400146+0000","last_peered":"2024-09-18T12:01:13.400146+0000","last_clean":"2024-09-18T12:01:13.400146+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:13.400146+0000","last_undegraded":"2024-09-18T12:01:13.400146+0000","last_fullsized":"2024-09-18T12:01:13.400146+0000","mapping_epoch":29,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21721088,"num_objects":5303,"num_object_clones":0,"num_object_copies":10606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11321,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14146","reported_seq":14333,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.502330+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:13.502330+0000","last_peered":"2024-09-18T12:01:13.502330+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:13.502330+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22163480,"num_objects":5412,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5412,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11004,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13960","reported_seq":13991,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:13.488674+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:13.488674+0000","last_peered":"2024-09-18T12:01:13.488674+0000","last_clean":"2024-09-18T12:01:13.488674+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:13.488674+0000","last_undegraded":"2024-09-18T12:01:13.488674+0000","last_fullsized":"2024-09-18T12:01:13.488674+0000","mapping_epoch":29,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21708800,"num_objects":5300,"num_object_clones":0,"num_object_copies":10600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13960,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14028","reported_seq":14221,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:13.286693+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:13.286693+0000","last_peered":"2024-09-18T12:01:13.286693+0000","last_clean":"2024-09-18T12:01:13.286693+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:13.286693+0000","last_undegraded":"2024-09-18T12:01:13.286693+0000","last_fullsized":"2024-09-18T12:01:13.286693+0000","mapping_epoch":29,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22196224,"num_objects":5419,"num_object_clones":0,"num_object_copies":10838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14029","reported_seq":14112,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.409730+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:13.409730+0000","last_peered":"2024-09-18T12:01:13.409730+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:13.409730+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22095058,"num_objects":5396,"num_object_clones":0,"num_object_copies":10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14027,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1869}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14049","reported_seq":14138,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.411952+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:13.411952+0000","last_peered":"2024-09-18T12:01:13.411952+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:13.411952+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21804242,"num_objects":5325,"num_object_clones":0,"num_object_copies":10650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14047,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1766}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14282","reported_seq":14509,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:13.356049+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:13.356049+0000","last_peered":"2024-09-18T12:01:13.356049+0000","last_clean":"2024-09-18T12:01:13.356049+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:13.356049+0000","last_undegraded":"2024-09-18T12:01:13.356049+0000","last_fullsized":"2024-09-18T12:01:13.356049+0000","mapping_epoch":29,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21790720,"num_objects":5320,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10599,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14026","reported_seq":21156,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:14.346061+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:14.346061+0000","last_peered":"2024-09-18T12:01:14.346061+0000","last_clean":"2024-09-18T12:01:14.346061+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:14.346061+0000","last_undegraded":"2024-09-18T12:01:14.346061+0000","last_fullsized":"2024-09-18T12:01:14.346061+0000","mapping_epoch":38,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22033618,"num_objects":5381,"num_object_clones":0,"num_object_copies":10762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14024,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14079","reported_seq":14159,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.410688+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:13.410688+0000","last_peered":"2024-09-18T12:01:13.410688+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:13.410688+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21984466,"num_objects":5369,"num_object_clones":0,"num_object_copies":10738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14077,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1837}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14109","reported_seq":14167,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.487497+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:13.487497+0000","last_peered":"2024-09-18T12:01:13.487497+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:13.487497+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21931218,"num_objects":5356,"num_object_clones":0,"num_object_copies":10712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14107,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1806}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14180","reported_seq":19630,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.501638+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:13.501638+0000","last_peered":"2024-09-18T12:01:13.501638+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:13.501638+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21782528,"num_objects":5318,"num_object_clones":0,"num_object_copies":10636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10660,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2619,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4342}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14149","reported_seq":14233,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.347697+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:13.347697+0000","last_peered":"2024-09-18T12:01:13.347697+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:13.347697+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22033001,"num_objects":5380,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14148,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1845}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14184","reported_seq":21332,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:14.346866+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:14.346866+0000","last_peered":"2024-09-18T12:01:14.346866+0000","last_clean":"2024-09-18T12:01:14.346866+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:14.346866+0000","last_undegraded":"2024-09-18T12:01:14.346866+0000","last_fullsized":"2024-09-18T12:01:14.346866+0000","mapping_epoch":40,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21866299,"num_objects":5341,"num_object_clones":0,"num_object_copies":10682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14181,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13990","reported_seq":14072,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.431338+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:13.431338+0000","last_peered":"2024-09-18T12:01:13.431338+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:13.431338+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21333202,"num_objects":5210,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13988,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14144","reported_seq":14231,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.387865+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:13.387865+0000","last_peered":"2024-09-18T12:01:13.387865+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:13.387865+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21989796,"num_objects":5372,"num_object_clones":0,"num_object_copies":10744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14141,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1849}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":350163020,"num_objects":85507,"num_object_clones":0,"num_object_copies":171014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85507,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":209708,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9647,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":873910272,"data_stored":873839506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160815,"ondisk_log_size":160815,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051672,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595728,"kb_used_data":285384,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93776112,"statfs":{"total":96636764160,"available":96026738688,"internally_reserved":0,"allocated":292233216,"data_stored":291995100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":6.1420000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":6.2080000000000002},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.33300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855435,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558124,"kb_used_data":256100,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93813716,"statfs":{"total":96636764160,"available":96065245184,"internally_reserved":0,"allocated":262246400,"data_stored":261952477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":1.0029999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":6.6619999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.30599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619396,"kb_used_data":313276,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93752444,"statfs":{"total":96636764160,"available":96002502656,"internally_reserved":0,"allocated":320794624,"data_stored":320556958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,2,9,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.17599999999999999},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.21199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":4.6470000000000002},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320380928,"data_stored":320377449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261849088,"data_stored":261789352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":291680256,"data_stored":291672705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:17.287 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:18.046 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:18.057 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:18.087 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-09-18T12:01:17.192961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":350163039,"num_objects":85509,"num_object_clones":0,"num_object_copies":171018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":209710,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9648,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160818,"ondisk_log_size":160818,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1773248,"kb_used_data":854760,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281342272,"statfs":{"total":289910292480,"available":288094486528,"internally_reserved":0,"allocated":875274240,"data_stored":874504535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,5,13,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17510400,"num_objects":-4275,"num_object_clones":0,"num_object_copies":-8550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4275,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002729"},"pg_stats":[{"pgid":"2.f","version":"41'14231","reported_seq":14302,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.450333+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:13.450333+0000","last_peered":"2024-09-18T12:01:13.450333+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:13.450333+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21729280,"num_objects":5305,"num_object_clones":0,"num_object_copies":10610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14231,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1706}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14129","reported_seq":14289,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:13.400146+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:13.400146+0000","last_peered":"2024-09-18T12:01:13.400146+0000","last_clean":"2024-09-18T12:01:13.400146+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:13.400146+0000","last_undegraded":"2024-09-18T12:01:13.400146+0000","last_fullsized":"2024-09-18T12:01:13.400146+0000","mapping_epoch":29,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21721088,"num_objects":5303,"num_object_clones":0,"num_object_copies":10606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11321,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14146","reported_seq":14333,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.502330+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:13.502330+0000","last_peered":"2024-09-18T12:01:13.502330+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:13.502330+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22163480,"num_objects":5412,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5412,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11004,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1876}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'13960","reported_seq":13991,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:13.488674+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:13.488674+0000","last_peered":"2024-09-18T12:01:13.488674+0000","last_clean":"2024-09-18T12:01:13.488674+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:13.488674+0000","last_undegraded":"2024-09-18T12:01:13.488674+0000","last_fullsized":"2024-09-18T12:01:13.488674+0000","mapping_epoch":29,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21708800,"num_objects":5300,"num_object_clones":0,"num_object_copies":10600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13960,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14028","reported_seq":14221,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:13.286693+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:13.286693+0000","last_peered":"2024-09-18T12:01:13.286693+0000","last_clean":"2024-09-18T12:01:13.286693+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:13.286693+0000","last_undegraded":"2024-09-18T12:01:13.286693+0000","last_fullsized":"2024-09-18T12:01:13.286693+0000","mapping_epoch":29,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22196224,"num_objects":5419,"num_object_clones":0,"num_object_copies":10838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14029","reported_seq":14112,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.409730+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:13.409730+0000","last_peered":"2024-09-18T12:01:13.409730+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:13.409730+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22095058,"num_objects":5396,"num_object_clones":0,"num_object_copies":10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14027,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1869}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14049","reported_seq":14138,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.411952+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:13.411952+0000","last_peered":"2024-09-18T12:01:13.411952+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:13.411952+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21804242,"num_objects":5325,"num_object_clones":0,"num_object_copies":10650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14047,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1766}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14282","reported_seq":14509,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:13.356049+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:13.356049+0000","last_peered":"2024-09-18T12:01:13.356049+0000","last_clean":"2024-09-18T12:01:13.356049+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:13.356049+0000","last_undegraded":"2024-09-18T12:01:13.356049+0000","last_fullsized":"2024-09-18T12:01:13.356049+0000","mapping_epoch":29,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21790720,"num_objects":5320,"num_object_clones":0,"num_object_copies":10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10599,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14026","reported_seq":21156,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:14.346061+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:14.346061+0000","last_peered":"2024-09-18T12:01:14.346061+0000","last_clean":"2024-09-18T12:01:14.346061+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:14.346061+0000","last_undegraded":"2024-09-18T12:01:14.346061+0000","last_fullsized":"2024-09-18T12:01:14.346061+0000","mapping_epoch":38,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22033618,"num_objects":5381,"num_object_clones":0,"num_object_copies":10762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14024,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14079","reported_seq":14159,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.410688+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:13.410688+0000","last_peered":"2024-09-18T12:01:13.410688+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:13.410688+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21984466,"num_objects":5369,"num_object_clones":0,"num_object_copies":10738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14077,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1837}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14109","reported_seq":14167,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.487497+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:13.487497+0000","last_peered":"2024-09-18T12:01:13.487497+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:13.487497+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21931218,"num_objects":5356,"num_object_clones":0,"num_object_copies":10712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14107,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1806}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14180","reported_seq":19630,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.501638+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:13.501638+0000","last_peered":"2024-09-18T12:01:13.501638+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:13.501638+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21782528,"num_objects":5318,"num_object_clones":0,"num_object_copies":10636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10660,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2619,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4342}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14149","reported_seq":14233,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.347697+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:13.347697+0000","last_peered":"2024-09-18T12:01:13.347697+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:13.347697+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22033001,"num_objects":5380,"num_object_clones":0,"num_object_copies":10760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14148,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1845}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14184","reported_seq":21332,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:14.346866+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:14.346866+0000","last_peered":"2024-09-18T12:01:14.346866+0000","last_clean":"2024-09-18T12:01:14.346866+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:14.346866+0000","last_undegraded":"2024-09-18T12:01:14.346866+0000","last_fullsized":"2024-09-18T12:01:14.346866+0000","mapping_epoch":40,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21866299,"num_objects":5341,"num_object_clones":0,"num_object_copies":10682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14181,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'13990","reported_seq":14072,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.431338+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:13.431338+0000","last_peered":"2024-09-18T12:01:13.431338+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:13.431338+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'3900","ondisk_log_start":"37'3900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21333202,"num_objects":5210,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13988,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14144","reported_seq":14231,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:13.387865+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:13.387865+0000","last_peered":"2024-09-18T12:01:13.387865+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:13.387865+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21989796,"num_objects":5372,"num_object_clones":0,"num_object_copies":10744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14141,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1849}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":350163020,"num_objects":85507,"num_object_clones":0,"num_object_copies":171014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85507,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":209708,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9647,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":873910272,"data_stored":873839506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160815,"ondisk_log_size":160815,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051672,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595728,"kb_used_data":285384,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93776112,"statfs":{"total":96636764160,"available":96026738688,"internally_reserved":0,"allocated":292233216,"data_stored":291995100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":6.1420000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":6.2080000000000002},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.33300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855435,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558124,"kb_used_data":256100,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93813716,"statfs":{"total":96636764160,"available":96065245184,"internally_reserved":0,"allocated":262246400,"data_stored":261952477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":1.0029999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":6.6619999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.30599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":619396,"kb_used_data":313276,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93752444,"statfs":{"total":96636764160,"available":96002502656,"internally_reserved":0,"allocated":320794624,"data_stored":320556958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,2,9,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.17599999999999999},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.21199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":4.6470000000000002},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320380928,"data_stored":320377449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261849088,"data_stored":261789352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":291680256,"data_stored":291672705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:18.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:01:18.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'14231", 'reported_seq': 14302, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:13.450333+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:01:13.450333+0000', 'last_peered': '2024-09-18T12:01:13.450333+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:01:13.450333+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'4200", 'ondisk_log_start': "37'4200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21729280, 'num_objects': 5305, 'num_object_clones': 0, 'num_object_copies': 10610, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14231, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1706}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:18.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:01:18.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'14146", 'reported_seq': 14333, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:13.502330+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:01:13.502330+0000', 'last_peered': '2024-09-18T12:01:13.502330+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:01:13.502330+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'4100", 'ondisk_log_start': "37'4100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22163480, 'num_objects': 5412, 'num_object_clones': 0, 'num_object_copies': 10824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5412, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11004, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1876}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:18.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:01:18.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'14029", 'reported_seq': 14112, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:13.409730+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:01:13.409730+0000', 'last_peered': '2024-09-18T12:01:13.409730+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:01:13.409730+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'4000", 'ondisk_log_start': "37'4000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22095058, 'num_objects': 5396, 'num_object_clones': 0, 'num_object_copies': 10792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5396, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14027, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1869}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:18.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:01:18.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'14049", 'reported_seq': 14138, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:13.411952+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:01:13.411952+0000', 'last_peered': '2024-09-18T12:01:13.411952+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:01:13.411952+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'4000", 'ondisk_log_start': "37'4000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21804242, 'num_objects': 5325, 'num_object_clones': 0, 'num_object_copies': 10650, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5325, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14047, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1766}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:18.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:01:18.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'14079", 'reported_seq': 14159, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:13.410688+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:01:13.410688+0000', 'last_peered': '2024-09-18T12:01:13.410688+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:01:13.410688+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'4000", 'ondisk_log_start': "37'4000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21984466, 'num_objects': 5369, 'num_object_clones': 0, 'num_object_copies': 10738, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14077, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1837}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:18.090 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:01:18.090 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'14109", 'reported_seq': 14167, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:13.487497+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:01:13.487497+0000', 'last_peered': '2024-09-18T12:01:13.487497+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:01:13.487497+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'4100", 'ondisk_log_start': "37'4100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21931218, 'num_objects': 5356, 'num_object_clones': 0, 'num_object_copies': 10712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14107, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1806}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:18.090 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:01:18.090 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'14180", 'reported_seq': 19630, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:13.501638+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:01:13.501638+0000', 'last_peered': '2024-09-18T12:01:13.501638+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:01:13.501638+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "37'4100", 'ondisk_log_start': "37'4100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21782528, 'num_objects': 5318, 'num_object_clones': 0, 'num_object_copies': 10636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10660, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2619, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 4342}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:18.090 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:01:18.090 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'14149", 'reported_seq': 14233, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:13.347697+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:01:13.347697+0000', 'last_peered': '2024-09-18T12:01:13.347697+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:01:13.347697+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'4100", 'ondisk_log_start': "37'4100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22033001, 'num_objects': 5380, 'num_object_clones': 0, 'num_object_copies': 10760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14148, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1845}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:18.090 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:01:18.090 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'13990", 'reported_seq': 14072, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:13.431338+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:01:13.431338+0000', 'last_peered': '2024-09-18T12:01:13.431338+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:01:13.431338+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'3900", 'ondisk_log_start': "37'3900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21333202, 'num_objects': 5210, 'num_object_clones': 0, 'num_object_copies': 10420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13988, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1617}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:18.090 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:01:18.090 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'14144", 'reported_seq': 14231, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:13.387865+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:01:13.387865+0000', 'last_peered': '2024-09-18T12:01:13.387865+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:01:13.387865+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'4100", 'ondisk_log_start': "37'4100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21989796, 'num_objects': 5372, 'num_object_clones': 0, 'num_object_copies': 10744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14141, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1849}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:18.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:01:18.927 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:18.927 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32945/171018 objects degraded (19.264%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":478,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":85509,"data_bytes":350163039,"bytes_used":1815805952,"bytes_avail":288094486528,"bytes_total":289910292480,"degraded_objects":32945,"degraded_total":171018,"degraded_ratio":0.19264054076179116,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":279,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":356},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:01:18.955 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:01:18.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:19.815 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:19.836 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:19.857 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-18T12:01:19.195516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":342384735,"num_objects":83610,"num_object_clones":0,"num_object_copies":167220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83610,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211609,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9801,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160817,"ondisk_log_size":160817,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1753848,"kb_used_data":843672,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281361672,"statfs":{"total":289910292480,"available":288114352128,"internally_reserved":0,"allocated":863920128,"data_stored":863150423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,8,8,7,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25288704,"num_objects":-6174,"num_object_clones":0,"num_object_copies":-12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6174,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":432,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004813"},"pg_stats":[{"pgid":"2.f","version":"41'14353","reported_seq":14426,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.407499+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:18.407499+0000","last_peered":"2024-09-18T12:01:18.407499+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:18.407499+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21229568,"num_objects":5183,"num_object_clones":0,"num_object_copies":10366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14353,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1584}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14264","reported_seq":14424,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:18.433453+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:18.433453+0000","last_peered":"2024-09-18T12:01:18.433453+0000","last_clean":"2024-09-18T12:01:18.433453+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:18.433453+0000","last_undegraded":"2024-09-18T12:01:18.433453+0000","last_fullsized":"2024-09-18T12:01:18.433453+0000","mapping_epoch":29,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21168128,"num_objects":5168,"num_object_clones":0,"num_object_copies":10336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11456,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14303","reported_seq":14490,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.396797+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:18.396797+0000","last_peered":"2024-09-18T12:01:18.396797+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:18.396797+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21520408,"num_objects":5255,"num_object_clones":0,"num_object_copies":10510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11161,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1719}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14111","reported_seq":14140,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:18.443949+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:18.443949+0000","last_peered":"2024-09-18T12:01:18.443949+0000","last_clean":"2024-09-18T12:01:18.443949+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:18.443949+0000","last_undegraded":"2024-09-18T12:01:18.443949+0000","last_fullsized":"2024-09-18T12:01:18.443949+0000","mapping_epoch":29,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21090304,"num_objects":5149,"num_object_clones":0,"num_object_copies":10298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14111,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14174","reported_seq":14367,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:18.445148+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:18.445148+0000","last_peered":"2024-09-18T12:01:18.445148+0000","last_clean":"2024-09-18T12:01:18.445148+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:18.445148+0000","last_undegraded":"2024-09-18T12:01:18.445148+0000","last_fullsized":"2024-09-18T12:01:18.445148+0000","mapping_epoch":29,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21598208,"num_objects":5273,"num_object_clones":0,"num_object_copies":10546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11339,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14153","reported_seq":14233,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.436739+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:18.436739+0000","last_peered":"2024-09-18T12:01:18.436739+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:18.436739+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21587154,"num_objects":5272,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14151,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14193","reported_seq":14282,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.374983+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:18.374983+0000","last_peered":"2024-09-18T12:01:18.374983+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:18.374983+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21214418,"num_objects":5181,"num_object_clones":0,"num_object_copies":10362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14191,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14436","reported_seq":14663,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:18.418903+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:18.418903+0000","last_peered":"2024-09-18T12:01:18.418903+0000","last_clean":"2024-09-18T12:01:18.418903+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:18.418903+0000","last_undegraded":"2024-09-18T12:01:18.418903+0000","last_fullsized":"2024-09-18T12:01:18.418903+0000","mapping_epoch":29,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21159936,"num_objects":5166,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10753,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14026","reported_seq":21156,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:14.346061+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:14.346061+0000","last_peered":"2024-09-18T12:01:14.346061+0000","last_clean":"2024-09-18T12:01:14.346061+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:14.346061+0000","last_undegraded":"2024-09-18T12:01:14.346061+0000","last_fullsized":"2024-09-18T12:01:14.346061+0000","mapping_epoch":38,"log_start":"37'4000","ondisk_log_start":"37'4000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22033618,"num_objects":5381,"num_object_clones":0,"num_object_copies":10762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14024,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14198","reported_seq":14274,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.437851+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:18.437851+0000","last_peered":"2024-09-18T12:01:18.437851+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:18.437851+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21497042,"num_objects":5250,"num_object_clones":0,"num_object_copies":10500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14196,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1718}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14239","reported_seq":14297,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.447600+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:18.447600+0000","last_peered":"2024-09-18T12:01:18.447600+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:18.447600+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21398738,"num_objects":5226,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14237,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1676}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14303","reported_seq":20059,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.518724+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:18.518724+0000","last_peered":"2024-09-18T12:01:18.518724+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:18.518724+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21278720,"num_objects":5195,"num_object_clones":0,"num_object_copies":10390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10783,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2772,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14283","reported_seq":14367,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.441478+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:18.441478+0000","last_peered":"2024-09-18T12:01:18.441478+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:18.441478+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21484137,"num_objects":5246,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14282,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1711}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14184","reported_seq":21332,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:14.346866+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:14.346866+0000","last_peered":"2024-09-18T12:01:14.346866+0000","last_clean":"2024-09-18T12:01:14.346866+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:14.346866+0000","last_undegraded":"2024-09-18T12:01:14.346866+0000","last_fullsized":"2024-09-18T12:01:14.346866+0000","mapping_epoch":40,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21866299,"num_objects":5341,"num_object_clones":0,"num_object_copies":10682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14181,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14111","reported_seq":14193,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.435380+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:18.435380+0000","last_peered":"2024-09-18T12:01:18.435380+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:18.435380+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20837586,"num_objects":5089,"num_object_clones":0,"num_object_copies":10178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14109,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14283","reported_seq":14370,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.375770+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:18.375770+0000","last_peered":"2024-09-18T12:01:18.375770+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:18.375770+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21420452,"num_objects":5233,"num_object_clones":0,"num_object_copies":10466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14280,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":342384716,"num_objects":83608,"num_object_clones":0,"num_object_copies":167216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83608,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211607,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9800,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":862502912,"data_stored":862432146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160814,"ondisk_log_size":160814,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051673,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578800,"kb_used_data":280928,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93793040,"statfs":{"total":96636764160,"available":96044072960,"internally_reserved":0,"allocated":287670272,"data_stored":287432156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":6.1420000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":6.2080000000000002},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.33300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855435,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558124,"kb_used_data":256100,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93813716,"statfs":{"total":96636764160,"available":96065245184,"internally_reserved":0,"allocated":262246400,"data_stored":261952477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":1.0029999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":6.6619999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.30599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616924,"kb_used_data":306644,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93754916,"statfs":{"total":96636764160,"available":96005033984,"internally_reserved":0,"allocated":314003456,"data_stored":313765790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,4,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":2.98},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.21099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313581568,"data_stored":313578089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261849088,"data_stored":261789352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287072256,"data_stored":287064705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:22.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:23.519 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:23.548 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:23.568 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-18T12:01:23.200474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":341196895,"num_objects":83320,"num_object_clones":0,"num_object_copies":166640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83320,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211899,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9801,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160807,"ondisk_log_size":160807,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1755680,"kb_used_data":837180,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281359840,"statfs":{"total":289910292480,"available":288112476160,"internally_reserved":0,"allocated":857272320,"data_stored":856502615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,8,11,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17866752,"num_objects":-4362,"num_object_clones":0,"num_object_copies":-8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4362,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008848"},"pg_stats":[{"pgid":"2.f","version":"41'14353","reported_seq":14426,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.407499+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:18.407499+0000","last_peered":"2024-09-18T12:01:18.407499+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:18.407499+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21229568,"num_objects":5183,"num_object_clones":0,"num_object_copies":10366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14353,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1584}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14264","reported_seq":14424,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:18.433453+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:18.433453+0000","last_peered":"2024-09-18T12:01:18.433453+0000","last_clean":"2024-09-18T12:01:18.433453+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:18.433453+0000","last_undegraded":"2024-09-18T12:01:18.433453+0000","last_fullsized":"2024-09-18T12:01:18.433453+0000","mapping_epoch":29,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21168128,"num_objects":5168,"num_object_clones":0,"num_object_copies":10336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11456,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14303","reported_seq":14490,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.396797+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:18.396797+0000","last_peered":"2024-09-18T12:01:18.396797+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:18.396797+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21520408,"num_objects":5255,"num_object_clones":0,"num_object_copies":10510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11161,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1719}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14111","reported_seq":14140,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:18.443949+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:18.443949+0000","last_peered":"2024-09-18T12:01:18.443949+0000","last_clean":"2024-09-18T12:01:18.443949+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:18.443949+0000","last_undegraded":"2024-09-18T12:01:18.443949+0000","last_fullsized":"2024-09-18T12:01:18.443949+0000","mapping_epoch":29,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21090304,"num_objects":5149,"num_object_clones":0,"num_object_copies":10298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14111,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14174","reported_seq":14367,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:18.445148+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:18.445148+0000","last_peered":"2024-09-18T12:01:18.445148+0000","last_clean":"2024-09-18T12:01:18.445148+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:18.445148+0000","last_undegraded":"2024-09-18T12:01:18.445148+0000","last_fullsized":"2024-09-18T12:01:18.445148+0000","mapping_epoch":29,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21598208,"num_objects":5273,"num_object_clones":0,"num_object_copies":10546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11339,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14153","reported_seq":14233,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.436739+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:18.436739+0000","last_peered":"2024-09-18T12:01:18.436739+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:18.436739+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21587154,"num_objects":5272,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14151,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14193","reported_seq":14282,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.374983+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:18.374983+0000","last_peered":"2024-09-18T12:01:18.374983+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:18.374983+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21214418,"num_objects":5181,"num_object_clones":0,"num_object_copies":10362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14191,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14436","reported_seq":14663,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:18.418903+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:18.418903+0000","last_peered":"2024-09-18T12:01:18.418903+0000","last_clean":"2024-09-18T12:01:18.418903+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:18.418903+0000","last_undegraded":"2024-09-18T12:01:18.418903+0000","last_fullsized":"2024-09-18T12:01:18.418903+0000","mapping_epoch":29,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21159936,"num_objects":5166,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10753,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14161","reported_seq":21291,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:19.365439+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:19.365439+0000","last_peered":"2024-09-18T12:01:19.365439+0000","last_clean":"2024-09-18T12:01:19.365439+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:19.365439+0000","last_undegraded":"2024-09-18T12:01:19.365439+0000","last_fullsized":"2024-09-18T12:01:19.365439+0000","mapping_epoch":38,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21480658,"num_objects":5246,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14159,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14198","reported_seq":14274,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.437851+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:18.437851+0000","last_peered":"2024-09-18T12:01:18.437851+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:18.437851+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21497042,"num_objects":5250,"num_object_clones":0,"num_object_copies":10500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14196,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1718}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14239","reported_seq":14297,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.447600+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:18.447600+0000","last_peered":"2024-09-18T12:01:18.447600+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:18.447600+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21398738,"num_objects":5226,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14237,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1676}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14303","reported_seq":20059,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.518724+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:18.518724+0000","last_peered":"2024-09-18T12:01:18.518724+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:18.518724+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21278720,"num_objects":5195,"num_object_clones":0,"num_object_copies":10390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10783,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2772,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14283","reported_seq":14367,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.441478+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:18.441478+0000","last_peered":"2024-09-18T12:01:18.441478+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:18.441478+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21484137,"num_objects":5246,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14282,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1711}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14339","reported_seq":21487,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:19.338789+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:19.338789+0000","last_peered":"2024-09-18T12:01:19.338789+0000","last_clean":"2024-09-18T12:01:19.338789+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:19.338789+0000","last_undegraded":"2024-09-18T12:01:19.338789+0000","last_fullsized":"2024-09-18T12:01:19.338789+0000","mapping_epoch":40,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21231419,"num_objects":5186,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14336,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14111","reported_seq":14193,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.435380+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:18.435380+0000","last_peered":"2024-09-18T12:01:18.435380+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:18.435380+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20837586,"num_objects":5089,"num_object_clones":0,"num_object_copies":10178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14109,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14283","reported_seq":14370,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.375770+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:18.375770+0000","last_peered":"2024-09-18T12:01:18.375770+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:18.375770+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21420452,"num_objects":5233,"num_object_clones":0,"num_object_copies":10466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14280,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":341196876,"num_objects":83318,"num_object_clones":0,"num_object_copies":166636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83318,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211897,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9800,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":855896064,"data_stored":855825298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160804,"ondisk_log_size":160804,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051673,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578800,"kb_used_data":280928,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93793040,"statfs":{"total":96636764160,"available":96044072960,"internally_reserved":0,"allocated":287670272,"data_stored":287432156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":6.1420000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":6.2080000000000002},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.33300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855436,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559956,"kb_used_data":249608,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93811884,"statfs":{"total":96636764160,"available":96063369216,"internally_reserved":0,"allocated":255598592,"data_stored":255304669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.193}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.254},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.373}]}]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616924,"kb_used_data":306644,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93754916,"statfs":{"total":96636764160,"available":96005033984,"internally_reserved":0,"allocated":314003456,"data_stored":313765790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,4,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":2.98},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.21099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313581568,"data_stored":313578089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255242240,"data_stored":255182504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287072256,"data_stored":287064705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:23.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:24.392 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:24.409 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:24.421 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-18T12:01:23.200474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":341196895,"num_objects":83320,"num_object_clones":0,"num_object_copies":166640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83320,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211899,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9801,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160807,"ondisk_log_size":160807,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1755680,"kb_used_data":837180,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281359840,"statfs":{"total":289910292480,"available":288112476160,"internally_reserved":0,"allocated":857272320,"data_stored":856502615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,8,11,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17866752,"num_objects":-4362,"num_object_clones":0,"num_object_copies":-8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4362,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008848"},"pg_stats":[{"pgid":"2.f","version":"41'14353","reported_seq":14426,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.407499+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:18.407499+0000","last_peered":"2024-09-18T12:01:18.407499+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:18.407499+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21229568,"num_objects":5183,"num_object_clones":0,"num_object_copies":10366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14353,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1584}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14264","reported_seq":14424,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:18.433453+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:18.433453+0000","last_peered":"2024-09-18T12:01:18.433453+0000","last_clean":"2024-09-18T12:01:18.433453+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:18.433453+0000","last_undegraded":"2024-09-18T12:01:18.433453+0000","last_fullsized":"2024-09-18T12:01:18.433453+0000","mapping_epoch":29,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21168128,"num_objects":5168,"num_object_clones":0,"num_object_copies":10336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11456,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14303","reported_seq":14490,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.396797+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:18.396797+0000","last_peered":"2024-09-18T12:01:18.396797+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:18.396797+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21520408,"num_objects":5255,"num_object_clones":0,"num_object_copies":10510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11161,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1719}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14111","reported_seq":14140,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:18.443949+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:18.443949+0000","last_peered":"2024-09-18T12:01:18.443949+0000","last_clean":"2024-09-18T12:01:18.443949+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:18.443949+0000","last_undegraded":"2024-09-18T12:01:18.443949+0000","last_fullsized":"2024-09-18T12:01:18.443949+0000","mapping_epoch":29,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21090304,"num_objects":5149,"num_object_clones":0,"num_object_copies":10298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14111,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14174","reported_seq":14367,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:18.445148+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:18.445148+0000","last_peered":"2024-09-18T12:01:18.445148+0000","last_clean":"2024-09-18T12:01:18.445148+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:18.445148+0000","last_undegraded":"2024-09-18T12:01:18.445148+0000","last_fullsized":"2024-09-18T12:01:18.445148+0000","mapping_epoch":29,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21598208,"num_objects":5273,"num_object_clones":0,"num_object_copies":10546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11339,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14153","reported_seq":14233,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.436739+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:18.436739+0000","last_peered":"2024-09-18T12:01:18.436739+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:18.436739+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21587154,"num_objects":5272,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14151,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14193","reported_seq":14282,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.374983+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:18.374983+0000","last_peered":"2024-09-18T12:01:18.374983+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:18.374983+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21214418,"num_objects":5181,"num_object_clones":0,"num_object_copies":10362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14191,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14436","reported_seq":14663,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:18.418903+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:18.418903+0000","last_peered":"2024-09-18T12:01:18.418903+0000","last_clean":"2024-09-18T12:01:18.418903+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:18.418903+0000","last_undegraded":"2024-09-18T12:01:18.418903+0000","last_fullsized":"2024-09-18T12:01:18.418903+0000","mapping_epoch":29,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21159936,"num_objects":5166,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10753,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14161","reported_seq":21291,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:19.365439+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:19.365439+0000","last_peered":"2024-09-18T12:01:19.365439+0000","last_clean":"2024-09-18T12:01:19.365439+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:19.365439+0000","last_undegraded":"2024-09-18T12:01:19.365439+0000","last_fullsized":"2024-09-18T12:01:19.365439+0000","mapping_epoch":38,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21480658,"num_objects":5246,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14159,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14198","reported_seq":14274,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.437851+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:18.437851+0000","last_peered":"2024-09-18T12:01:18.437851+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:18.437851+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21497042,"num_objects":5250,"num_object_clones":0,"num_object_copies":10500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14196,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1718}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14239","reported_seq":14297,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.447600+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:18.447600+0000","last_peered":"2024-09-18T12:01:18.447600+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:18.447600+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21398738,"num_objects":5226,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14237,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1676}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14303","reported_seq":20059,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.518724+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:18.518724+0000","last_peered":"2024-09-18T12:01:18.518724+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:18.518724+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21278720,"num_objects":5195,"num_object_clones":0,"num_object_copies":10390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":823,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10783,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2772,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14283","reported_seq":14367,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.441478+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:18.441478+0000","last_peered":"2024-09-18T12:01:18.441478+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:18.441478+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21484137,"num_objects":5246,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14282,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1711}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14339","reported_seq":21487,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:19.338789+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:19.338789+0000","last_peered":"2024-09-18T12:01:19.338789+0000","last_clean":"2024-09-18T12:01:19.338789+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:19.338789+0000","last_undegraded":"2024-09-18T12:01:19.338789+0000","last_fullsized":"2024-09-18T12:01:19.338789+0000","mapping_epoch":40,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21231419,"num_objects":5186,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14336,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14111","reported_seq":14193,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.435380+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:18.435380+0000","last_peered":"2024-09-18T12:01:18.435380+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:18.435380+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4100","ondisk_log_start":"37'4100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20837586,"num_objects":5089,"num_object_clones":0,"num_object_copies":10178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14109,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14283","reported_seq":14370,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:18.375770+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:18.375770+0000","last_peered":"2024-09-18T12:01:18.375770+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:18.375770+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21420452,"num_objects":5233,"num_object_clones":0,"num_object_copies":10466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14280,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":341196876,"num_objects":83318,"num_object_clones":0,"num_object_copies":166636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83318,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211897,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9800,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":855896064,"data_stored":855825298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160804,"ondisk_log_size":160804,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051673,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578800,"kb_used_data":280928,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93793040,"statfs":{"total":96636764160,"available":96044072960,"internally_reserved":0,"allocated":287670272,"data_stored":287432156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":6.1420000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":6.2080000000000002},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.33300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855436,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559956,"kb_used_data":249608,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93811884,"statfs":{"total":96636764160,"available":96063369216,"internally_reserved":0,"allocated":255598592,"data_stored":255304669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.193}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.254},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.373}]}]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616924,"kb_used_data":306644,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93754916,"statfs":{"total":96636764160,"available":96005033984,"internally_reserved":0,"allocated":314003456,"data_stored":313765790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,4,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:00:17 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.995,"15min":1.018},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":4.8959999999999999,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":2.98},{"interface":"front","average":{"1min":1.173,"5min":0.97199999999999998,"15min":1.0980000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":8.3460000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.21099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":313581568,"data_stored":313578089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255242240,"data_stored":255182504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":287072256,"data_stored":287064705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:24.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:01:24.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'14353", 'reported_seq': 14426, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:18.407499+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:01:18.407499+0000', 'last_peered': '2024-09-18T12:01:18.407499+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:01:18.407499+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'4300", 'ondisk_log_start': "37'4300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21229568, 'num_objects': 5183, 'num_object_clones': 0, 'num_object_copies': 10366, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14353, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1584}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:24.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:01:24.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'14303", 'reported_seq': 14490, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:18.396797+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:01:18.396797+0000', 'last_peered': '2024-09-18T12:01:18.396797+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:01:18.396797+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'4300", 'ondisk_log_start': "37'4300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21520408, 'num_objects': 5255, 'num_object_clones': 0, 'num_object_copies': 10510, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5255, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11161, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1719}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:24.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:01:24.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'14153", 'reported_seq': 14233, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:18.436739+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:01:18.436739+0000', 'last_peered': '2024-09-18T12:01:18.436739+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:01:18.436739+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'4100", 'ondisk_log_start': "37'4100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21587154, 'num_objects': 5272, 'num_object_clones': 0, 'num_object_copies': 10544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14151, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1745}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:24.424 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:01:24.424 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'14193", 'reported_seq': 14282, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:18.374983+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:01:18.374983+0000', 'last_peered': '2024-09-18T12:01:18.374983+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:01:18.374983+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'4100", 'ondisk_log_start': "37'4100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21214418, 'num_objects': 5181, 'num_object_clones': 0, 'num_object_copies': 10362, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5181, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14191, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1622}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:24.424 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:01:24.424 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'14198", 'reported_seq': 14274, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:18.437851+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:01:18.437851+0000', 'last_peered': '2024-09-18T12:01:18.437851+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:01:18.437851+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'4100", 'ondisk_log_start': "37'4100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21497042, 'num_objects': 5250, 'num_object_clones': 0, 'num_object_copies': 10500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14196, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1718}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:24.424 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:01:24.424 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'14239", 'reported_seq': 14297, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:18.447600+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:01:18.447600+0000', 'last_peered': '2024-09-18T12:01:18.447600+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:01:18.447600+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'4200", 'ondisk_log_start': "37'4200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21398738, 'num_objects': 5226, 'num_object_clones': 0, 'num_object_copies': 10452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14237, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1676}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:24.424 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:01:24.424 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'14303", 'reported_seq': 20059, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:18.518724+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:01:18.518724+0000', 'last_peered': '2024-09-18T12:01:18.518724+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:01:18.518724+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "37'4300", 'ondisk_log_start': "37'4300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21278720, 'num_objects': 5195, 'num_object_clones': 0, 'num_object_copies': 10390, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 823, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10783, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2772, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 4372}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:24.424 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:01:24.424 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'14283", 'reported_seq': 14367, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:18.441478+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:01:18.441478+0000', 'last_peered': '2024-09-18T12:01:18.441478+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:01:18.441478+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'4200", 'ondisk_log_start': "37'4200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21484137, 'num_objects': 5246, 'num_object_clones': 0, 'num_object_copies': 10492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5246, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14282, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1711}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:24.424 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:01:24.425 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'14111", 'reported_seq': 14193, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:18.435380+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:01:18.435380+0000', 'last_peered': '2024-09-18T12:01:18.435380+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:01:18.435380+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'4100", 'ondisk_log_start': "37'4100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20837586, 'num_objects': 5089, 'num_object_clones': 0, 'num_object_copies': 10178, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14109, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1496}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:24.425 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:01:24.425 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'14283", 'reported_seq': 14370, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:18.375770+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:01:18.375770+0000', 'last_peered': '2024-09-18T12:01:18.375770+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:01:18.375770+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'4200", 'ondisk_log_start': "37'4200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21420452, 'num_objects': 5233, 'num_object_clones': 0, 'num_object_copies': 10466, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14280, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1710}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:24.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:01:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:25.379 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32792/166640 objects degraded (19.678%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":485,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":83320,"data_bytes":341196895,"bytes_used":1797816320,"bytes_avail":288112476160,"bytes_total":289910292480,"degraded_objects":32792,"degraded_total":166640,"degraded_ratio":0.19678348535765722,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":287,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":363},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:01:25.438 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:01:25.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:26.152 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:26.202 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:26.207 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-18T12:01:25.200897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":332320863,"num_objects":81153,"num_object_clones":0,"num_object_copies":162306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81153,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":214066,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9967,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160874,"ondisk_log_size":160874,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1725784,"kb_used_data":819772,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281389736,"statfs":{"total":289910292480,"available":288143089664,"internally_reserved":0,"allocated":839446528,"data_stored":838676823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,8,7,12,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26742784,"num_objects":-6529,"num_object_clones":0,"num_object_copies":-13058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6529,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":453,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008820"},"pg_stats":[{"pgid":"2.f","version":"41'14489","reported_seq":14562,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:23.461229+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:23.461229+0000","last_peered":"2024-09-18T12:01:23.461229+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:23.461229+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20672512,"num_objects":5047,"num_object_clones":0,"num_object_copies":10094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14489,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1448}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14395","reported_seq":14555,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:23.443037+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:23.443037+0000","last_peered":"2024-09-18T12:01:23.443037+0000","last_clean":"2024-09-18T12:01:23.443037+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:23.443037+0000","last_undegraded":"2024-09-18T12:01:23.443037+0000","last_fullsized":"2024-09-18T12:01:23.443037+0000","mapping_epoch":29,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20631552,"num_objects":5037,"num_object_clones":0,"num_object_copies":10074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11587,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14432","reported_seq":14620,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:23.353241+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:23.353241+0000","last_peered":"2024-09-18T12:01:23.353241+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:23.353241+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20992024,"num_objects":5126,"num_object_clones":0,"num_object_copies":10252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5126,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11290,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1590}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14230","reported_seq":14261,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:23.389931+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:23.389931+0000","last_peered":"2024-09-18T12:01:23.389931+0000","last_clean":"2024-09-18T12:01:23.389931+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:23.389931+0000","last_undegraded":"2024-09-18T12:01:23.389931+0000","last_fullsized":"2024-09-18T12:01:23.389931+0000","mapping_epoch":29,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20602880,"num_objects":5030,"num_object_clones":0,"num_object_copies":10060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14230,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14307","reported_seq":14500,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:23.412652+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:23.412652+0000","last_peered":"2024-09-18T12:01:23.412652+0000","last_clean":"2024-09-18T12:01:23.412652+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:23.412652+0000","last_undegraded":"2024-09-18T12:01:23.412652+0000","last_fullsized":"2024-09-18T12:01:23.412652+0000","mapping_epoch":29,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21053440,"num_objects":5140,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11472,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14297","reported_seq":14379,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:23.417364+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:23.417364+0000","last_peered":"2024-09-18T12:01:23.417364+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:23.417364+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20997330,"num_objects":5128,"num_object_clones":0,"num_object_copies":10256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14295,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14326","reported_seq":14415,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:23.451408+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:23.451408+0000","last_peered":"2024-09-18T12:01:23.451408+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:23.451408+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20669650,"num_objects":5048,"num_object_clones":0,"num_object_copies":10096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14324,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1489}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14587","reported_seq":14814,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:23.487473+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:23.487473+0000","last_peered":"2024-09-18T12:01:23.487473+0000","last_clean":"2024-09-18T12:01:23.487473+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:23.487473+0000","last_undegraded":"2024-09-18T12:01:23.487473+0000","last_fullsized":"2024-09-18T12:01:23.487473+0000","mapping_epoch":29,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20541440,"num_objects":5015,"num_object_clones":0,"num_object_copies":10030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10904,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14294","reported_seq":21424,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:24.353480+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:24.353480+0000","last_peered":"2024-09-18T12:01:24.353480+0000","last_clean":"2024-09-18T12:01:24.353480+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:24.353480+0000","last_undegraded":"2024-09-18T12:01:24.353480+0000","last_fullsized":"2024-09-18T12:01:24.353480+0000","mapping_epoch":38,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20935890,"num_objects":5113,"num_object_clones":0,"num_object_copies":10226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14292,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14336","reported_seq":14415,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:23.345893+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:23.345893+0000","last_peered":"2024-09-18T12:01:23.345893+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:23.345893+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20931794,"num_objects":5112,"num_object_clones":0,"num_object_copies":10224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14334,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1580}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14375","reported_seq":14433,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:23.522322+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:23.522322+0000","last_peered":"2024-09-18T12:01:23.522322+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:23.522322+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20841682,"num_objects":5090,"num_object_clones":0,"num_object_copies":10180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14373,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1540}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14417","reported_seq":20506,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:23.513194+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:23.513194+0000","last_peered":"2024-09-18T12:01:23.513194+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:23.513194+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20811776,"num_objects":5081,"num_object_clones":0,"num_object_copies":10162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10897,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2938,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4424}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14416","reported_seq":14500,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:23.455729+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:23.455729+0000","last_peered":"2024-09-18T12:01:23.455729+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:23.455729+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20939369,"num_objects":5113,"num_object_clones":0,"num_object_copies":10226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14415,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14484","reported_seq":21632,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:24.415299+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:24.415299+0000","last_peered":"2024-09-18T12:01:24.415299+0000","last_clean":"2024-09-18T12:01:24.415299+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:24.415299+0000","last_undegraded":"2024-09-18T12:01:24.415299+0000","last_fullsized":"2024-09-18T12:01:24.415299+0000","mapping_epoch":40,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20637499,"num_objects":5041,"num_object_clones":0,"num_object_copies":10082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14481,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14259","reported_seq":14339,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:23.492883+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:23.492883+0000","last_peered":"2024-09-18T12:01:23.492883+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:23.492883+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20231378,"num_objects":4941,"num_object_clones":0,"num_object_copies":9882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14257,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14427","reported_seq":14514,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:23.475279+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:23.475279+0000","last_peered":"2024-09-18T12:01:23.475279+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:23.475279+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20830628,"num_objects":5089,"num_object_clones":0,"num_object_copies":10178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14424,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":332320844,"num_objects":81151,"num_object_clones":0,"num_object_copies":162302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81151,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":214064,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9966,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":837939200,"data_stored":837868434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051674,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578668,"kb_used_data":276644,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93793172,"statfs":{"total":96636764160,"available":96044208128,"internally_reserved":0,"allocated":283283456,"data_stored":283045340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:00:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.77100000000000002,"15min":0.50900000000000001},"min":{"1min":0.17799999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.8689999999999998,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.33300000000000002},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.53,"15min":0.70399999999999996},"min":{"1min":0.252,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":9.9930000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.19400000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.373}]}]},{"osd":1,"up_from":35,"seq":150323855437,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545100,"kb_used_data":243076,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93826740,"statfs":{"total":96636764160,"available":96078581760,"internally_reserved":0,"allocated":248909824,"data_stored":248615901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.23200000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.40100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602016,"kb_used_data":300052,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93769824,"statfs":{"total":96636764160,"available":96020299776,"internally_reserved":0,"allocated":307253248,"data_stored":307015582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.28899999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":0.35199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306757632,"data_stored":306754153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248401920,"data_stored":248342184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":282779648,"data_stored":282772097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:29.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:29.893 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:29.918 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:29.975 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-18T12:01:29.201811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":324509791,"num_objects":79246,"num_object_clones":0,"num_object_copies":158492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215973,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10139,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160881,"ondisk_log_size":160881,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1723348,"kb_used_data":809020,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281392172,"statfs":{"total":289910292480,"available":288145584128,"internally_reserved":0,"allocated":828436480,"data_stored":827666775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,5,6,9,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25653248,"num_objects":-6263,"num_object_clones":0,"num_object_copies":-12526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6263,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008850"},"pg_stats":[{"pgid":"2.f","version":"41'14635","reported_seq":14704,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.439761+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:28.439761+0000","last_peered":"2024-09-18T12:01:28.439761+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:28.439761+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20074496,"num_objects":4901,"num_object_clones":0,"num_object_copies":9802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14635,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1302}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14509","reported_seq":14669,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:28.410642+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:28.410642+0000","last_peered":"2024-09-18T12:01:28.410642+0000","last_clean":"2024-09-18T12:01:28.410642+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:28.410642+0000","last_undegraded":"2024-09-18T12:01:28.410642+0000","last_fullsized":"2024-09-18T12:01:28.410642+0000","mapping_epoch":29,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20164608,"num_objects":4923,"num_object_clones":0,"num_object_copies":9846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11701,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14572","reported_seq":14760,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.506771+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:28.506771+0000","last_peered":"2024-09-18T12:01:28.506771+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:28.506771+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20418584,"num_objects":4986,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11430,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14390","reported_seq":14421,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:28.443439+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:28.443439+0000","last_peered":"2024-09-18T12:01:28.443439+0000","last_clean":"2024-09-18T12:01:28.443439+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:28.443439+0000","last_undegraded":"2024-09-18T12:01:28.443439+0000","last_fullsized":"2024-09-18T12:01:28.443439+0000","mapping_epoch":29,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19947520,"num_objects":4870,"num_object_clones":0,"num_object_copies":9740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14390,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14455","reported_seq":14648,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:28.294917+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:28.294917+0000","last_peered":"2024-09-18T12:01:28.294917+0000","last_clean":"2024-09-18T12:01:28.294917+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:28.294917+0000","last_undegraded":"2024-09-18T12:01:28.294917+0000","last_fullsized":"2024-09-18T12:01:28.294917+0000","mapping_epoch":29,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20447232,"num_objects":4992,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11620,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14427","reported_seq":14510,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.493148+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:28.493148+0000","last_peered":"2024-09-18T12:01:28.493148+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:28.493148+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20464850,"num_objects":4998,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14425,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14455","reported_seq":14544,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.458250+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:28.458250+0000","last_peered":"2024-09-18T12:01:28.458250+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:28.458250+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141266,"num_objects":4919,"num_object_clones":0,"num_object_copies":9838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14453,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1360}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14709","reported_seq":14936,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:28.506961+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:28.506961+0000","last_peered":"2024-09-18T12:01:28.506961+0000","last_clean":"2024-09-18T12:01:28.506961+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:28.506961+0000","last_undegraded":"2024-09-18T12:01:28.506961+0000","last_fullsized":"2024-09-18T12:01:28.506961+0000","mapping_epoch":29,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20041728,"num_objects":4893,"num_object_clones":0,"num_object_copies":9786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11026,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14294","reported_seq":21424,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:24.353480+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:24.353480+0000","last_peered":"2024-09-18T12:01:24.353480+0000","last_clean":"2024-09-18T12:01:24.353480+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:24.353480+0000","last_undegraded":"2024-09-18T12:01:24.353480+0000","last_fullsized":"2024-09-18T12:01:24.353480+0000","mapping_epoch":38,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20935890,"num_objects":5113,"num_object_clones":0,"num_object_copies":10226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14292,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14481","reported_seq":14561,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.445640+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:28.445640+0000","last_peered":"2024-09-18T12:01:28.445640+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:28.445640+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337874,"num_objects":4967,"num_object_clones":0,"num_object_copies":9934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14479,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1435}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14530","reported_seq":14588,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.451594+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:28.451594+0000","last_peered":"2024-09-18T12:01:28.451594+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:28.451594+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20206802,"num_objects":4935,"num_object_clones":0,"num_object_copies":9870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14528,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14550","reported_seq":20982,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.507720+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:28.507720+0000","last_peered":"2024-09-18T12:01:28.507720+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:28.507720+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20267008,"num_objects":4948,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":485,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11030,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3110,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4463}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14549","reported_seq":14631,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.481869+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:28.481869+0000","last_peered":"2024-09-18T12:01:28.481869+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:28.481869+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20394601,"num_objects":4980,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14548,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1445}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14484","reported_seq":21632,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:24.415299+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:24.415299+0000","last_peered":"2024-09-18T12:01:24.415299+0000","last_clean":"2024-09-18T12:01:24.415299+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:24.415299+0000","last_undegraded":"2024-09-18T12:01:24.415299+0000","last_fullsized":"2024-09-18T12:01:24.415299+0000","mapping_epoch":40,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20637499,"num_objects":5041,"num_object_clones":0,"num_object_copies":10082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14481,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14376","reported_seq":14459,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.504487+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:28.504487+0000","last_peered":"2024-09-18T12:01:28.504487+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:28.504487+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19752146,"num_objects":4824,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14374,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14562","reported_seq":14649,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.460823+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:28.460823+0000","last_peered":"2024-09-18T12:01:28.460823+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:28.460823+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20277668,"num_objects":4954,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14559,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":324509772,"num_objects":79244,"num_object_clones":0,"num_object_copies":158488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79244,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215971,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10138,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":826867712,"data_stored":826796946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160878,"ondisk_log_size":160878,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051675,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578432,"kb_used_data":272248,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93793408,"statfs":{"total":96636764160,"available":96044449792,"internally_reserved":0,"allocated":278781952,"data_stored":278543836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.24199999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.26700000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.434}]}]},{"osd":1,"up_from":35,"seq":150323855437,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545100,"kb_used_data":243076,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93826740,"statfs":{"total":96636764160,"available":96078581760,"internally_reserved":0,"allocated":248909824,"data_stored":248615901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.23200000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.40100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599816,"kb_used_data":293696,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93772024,"statfs":{"total":96636764160,"available":96022552576,"internally_reserved":0,"allocated":300744704,"data_stored":300507038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":1.74},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":2.294}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":2.5129999999999999},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":2.4609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":300240896,"data_stored":300237417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248401920,"data_stored":248342184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278224896,"data_stored":278217345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:29.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:30.694 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:30.705 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:30.726 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-18T12:01:29.201811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":324509791,"num_objects":79246,"num_object_clones":0,"num_object_copies":158492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215973,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10139,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160881,"ondisk_log_size":160881,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1723348,"kb_used_data":809020,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281392172,"statfs":{"total":289910292480,"available":288145584128,"internally_reserved":0,"allocated":828436480,"data_stored":827666775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,5,6,9,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25653248,"num_objects":-6263,"num_object_clones":0,"num_object_copies":-12526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6263,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008850"},"pg_stats":[{"pgid":"2.f","version":"41'14635","reported_seq":14704,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.439761+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:28.439761+0000","last_peered":"2024-09-18T12:01:28.439761+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:28.439761+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20074496,"num_objects":4901,"num_object_clones":0,"num_object_copies":9802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14635,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1302}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14509","reported_seq":14669,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:28.410642+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:28.410642+0000","last_peered":"2024-09-18T12:01:28.410642+0000","last_clean":"2024-09-18T12:01:28.410642+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:28.410642+0000","last_undegraded":"2024-09-18T12:01:28.410642+0000","last_fullsized":"2024-09-18T12:01:28.410642+0000","mapping_epoch":29,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20164608,"num_objects":4923,"num_object_clones":0,"num_object_copies":9846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11701,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14572","reported_seq":14760,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.506771+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:28.506771+0000","last_peered":"2024-09-18T12:01:28.506771+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:28.506771+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20418584,"num_objects":4986,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11430,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14390","reported_seq":14421,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:28.443439+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:28.443439+0000","last_peered":"2024-09-18T12:01:28.443439+0000","last_clean":"2024-09-18T12:01:28.443439+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:28.443439+0000","last_undegraded":"2024-09-18T12:01:28.443439+0000","last_fullsized":"2024-09-18T12:01:28.443439+0000","mapping_epoch":29,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19947520,"num_objects":4870,"num_object_clones":0,"num_object_copies":9740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14390,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14455","reported_seq":14648,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:28.294917+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:28.294917+0000","last_peered":"2024-09-18T12:01:28.294917+0000","last_clean":"2024-09-18T12:01:28.294917+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:28.294917+0000","last_undegraded":"2024-09-18T12:01:28.294917+0000","last_fullsized":"2024-09-18T12:01:28.294917+0000","mapping_epoch":29,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20447232,"num_objects":4992,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11620,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14427","reported_seq":14510,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.493148+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:28.493148+0000","last_peered":"2024-09-18T12:01:28.493148+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:28.493148+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20464850,"num_objects":4998,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14425,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14455","reported_seq":14544,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.458250+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:28.458250+0000","last_peered":"2024-09-18T12:01:28.458250+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:28.458250+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141266,"num_objects":4919,"num_object_clones":0,"num_object_copies":9838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14453,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1360}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14709","reported_seq":14936,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:28.506961+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:28.506961+0000","last_peered":"2024-09-18T12:01:28.506961+0000","last_clean":"2024-09-18T12:01:28.506961+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:28.506961+0000","last_undegraded":"2024-09-18T12:01:28.506961+0000","last_fullsized":"2024-09-18T12:01:28.506961+0000","mapping_epoch":29,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20041728,"num_objects":4893,"num_object_clones":0,"num_object_copies":9786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11026,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14294","reported_seq":21424,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:24.353480+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:24.353480+0000","last_peered":"2024-09-18T12:01:24.353480+0000","last_clean":"2024-09-18T12:01:24.353480+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:24.353480+0000","last_undegraded":"2024-09-18T12:01:24.353480+0000","last_fullsized":"2024-09-18T12:01:24.353480+0000","mapping_epoch":38,"log_start":"37'4200","ondisk_log_start":"37'4200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20935890,"num_objects":5113,"num_object_clones":0,"num_object_copies":10226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14292,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14481","reported_seq":14561,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.445640+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:28.445640+0000","last_peered":"2024-09-18T12:01:28.445640+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:28.445640+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337874,"num_objects":4967,"num_object_clones":0,"num_object_copies":9934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14479,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1435}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14530","reported_seq":14588,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.451594+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:28.451594+0000","last_peered":"2024-09-18T12:01:28.451594+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:28.451594+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20206802,"num_objects":4935,"num_object_clones":0,"num_object_copies":9870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14528,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14550","reported_seq":20982,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.507720+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:28.507720+0000","last_peered":"2024-09-18T12:01:28.507720+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:28.507720+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20267008,"num_objects":4948,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":485,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11030,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3110,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4463}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14549","reported_seq":14631,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.481869+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:28.481869+0000","last_peered":"2024-09-18T12:01:28.481869+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:28.481869+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20394601,"num_objects":4980,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14548,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1445}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14484","reported_seq":21632,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:24.415299+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:24.415299+0000","last_peered":"2024-09-18T12:01:24.415299+0000","last_clean":"2024-09-18T12:01:24.415299+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:24.415299+0000","last_undegraded":"2024-09-18T12:01:24.415299+0000","last_fullsized":"2024-09-18T12:01:24.415299+0000","mapping_epoch":40,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20637499,"num_objects":5041,"num_object_clones":0,"num_object_copies":10082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14481,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14376","reported_seq":14459,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.504487+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:28.504487+0000","last_peered":"2024-09-18T12:01:28.504487+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:28.504487+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19752146,"num_objects":4824,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14374,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14562","reported_seq":14649,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.460823+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:28.460823+0000","last_peered":"2024-09-18T12:01:28.460823+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:28.460823+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20277668,"num_objects":4954,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14559,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":324509772,"num_objects":79244,"num_object_clones":0,"num_object_copies":158488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79244,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215971,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10138,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":826867712,"data_stored":826796946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160878,"ondisk_log_size":160878,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051675,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578432,"kb_used_data":272248,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93793408,"statfs":{"total":96636764160,"available":96044449792,"internally_reserved":0,"allocated":278781952,"data_stored":278543836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.24199999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.26700000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.434}]}]},{"osd":1,"up_from":35,"seq":150323855437,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545100,"kb_used_data":243076,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93826740,"statfs":{"total":96636764160,"available":96078581760,"internally_reserved":0,"allocated":248909824,"data_stored":248615901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.23200000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.40100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599816,"kb_used_data":293696,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93772024,"statfs":{"total":96636764160,"available":96022552576,"internally_reserved":0,"allocated":300744704,"data_stored":300507038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":1.74},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":2.294}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":2.5129999999999999},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":2.4609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":300240896,"data_stored":300237417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":248401920,"data_stored":248342184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278224896,"data_stored":278217345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:30.727 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:01:30.727 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'14635", 'reported_seq': 14704, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:28.439761+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:01:28.439761+0000', 'last_peered': '2024-09-18T12:01:28.439761+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:01:28.439761+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'4600", 'ondisk_log_start': "37'4600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20074496, 'num_objects': 4901, 'num_object_clones': 0, 'num_object_copies': 9802, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14635, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1302}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:30.728 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:01:30.728 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'14572", 'reported_seq': 14760, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:28.506771+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:01:28.506771+0000', 'last_peered': '2024-09-18T12:01:28.506771+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:01:28.506771+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'4500", 'ondisk_log_start': "37'4500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20418584, 'num_objects': 4986, 'num_object_clones': 0, 'num_object_copies': 9972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4986, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11430, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1450}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:30.728 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:01:30.728 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'14427", 'reported_seq': 14510, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:28.493148+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:01:28.493148+0000', 'last_peered': '2024-09-18T12:01:28.493148+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:01:28.493148+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'4400", 'ondisk_log_start': "37'4400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20464850, 'num_objects': 4998, 'num_object_clones': 0, 'num_object_copies': 9996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4998, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14425, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1471}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:30.728 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:01:30.728 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'14455", 'reported_seq': 14544, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:28.458250+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:01:28.458250+0000', 'last_peered': '2024-09-18T12:01:28.458250+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:01:28.458250+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'4400", 'ondisk_log_start': "37'4400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20141266, 'num_objects': 4919, 'num_object_clones': 0, 'num_object_copies': 9838, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4919, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14453, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1360}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:30.728 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:01:30.728 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'14481", 'reported_seq': 14561, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:28.445640+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:01:28.445640+0000', 'last_peered': '2024-09-18T12:01:28.445640+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:01:28.445640+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'4400", 'ondisk_log_start': "37'4400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20337874, 'num_objects': 4967, 'num_object_clones': 0, 'num_object_copies': 9934, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14479, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1435}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:30.728 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:01:30.728 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'14530", 'reported_seq': 14588, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:28.451594+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:01:28.451594+0000', 'last_peered': '2024-09-18T12:01:28.451594+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:01:28.451594+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'4500", 'ondisk_log_start': "37'4500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20206802, 'num_objects': 4935, 'num_object_clones': 0, 'num_object_copies': 9870, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4935, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14528, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1385}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:30.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:01:30.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'14550", 'reported_seq': 20982, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:28.507720+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:01:28.507720+0000', 'last_peered': '2024-09-18T12:01:28.507720+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:01:28.507720+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "37'4500", 'ondisk_log_start': "37'4500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20267008, 'num_objects': 4948, 'num_object_clones': 0, 'num_object_copies': 9896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 485, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11030, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3110, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 4463}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:30.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:01:30.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'14549", 'reported_seq': 14631, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:28.481869+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:01:28.481869+0000', 'last_peered': '2024-09-18T12:01:28.481869+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:01:28.481869+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'4500", 'ondisk_log_start': "37'4500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20394601, 'num_objects': 4980, 'num_object_clones': 0, 'num_object_copies': 9960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14548, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1445}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:30.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:01:30.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'14376", 'reported_seq': 14459, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:28.504487+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:01:28.504487+0000', 'last_peered': '2024-09-18T12:01:28.504487+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:01:28.504487+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'4300", 'ondisk_log_start': "37'4300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19752146, 'num_objects': 4824, 'num_object_clones': 0, 'num_object_copies': 9648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4824, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14374, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:30.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:01:30.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'14562", 'reported_seq': 14649, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:28.460823+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:01:28.460823+0000', 'last_peered': '2024-09-18T12:01:28.460823+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:01:28.460823+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'4500", 'ondisk_log_start': "37'4500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20277668, 'num_objects': 4954, 'num_object_clones': 0, 'num_object_copies': 9908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14559, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1431}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:30.730 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:01:31.667 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:31.667 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32454/157934 objects degraded (20.549%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":491,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":78967,"data_bytes":323367007,"bytes_used":1762320384,"bytes_avail":288147972096,"bytes_total":289910292480,"degraded_objects":32454,"degraded_total":157934,"degraded_ratio":0.20549090126255268,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":338,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":386},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:01:31.705 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:01:31.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:32.339 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:32.386 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:32.392 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-18T12:01:31.202401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":323367007,"num_objects":78967,"num_object_clones":0,"num_object_copies":157934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78967,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":216252,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10139,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1721016,"kb_used_data":802528,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281394504,"statfs":{"total":289910292480,"available":288147972096,"internally_reserved":0,"allocated":821788672,"data_stored":821018967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,5,7,8,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19017728,"num_objects":-4643,"num_object_clones":0,"num_object_copies":-9286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4643,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006884"},"pg_stats":[{"pgid":"2.f","version":"41'14635","reported_seq":14704,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.439761+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:28.439761+0000","last_peered":"2024-09-18T12:01:28.439761+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:28.439761+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20074496,"num_objects":4901,"num_object_clones":0,"num_object_copies":9802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14635,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1302}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14509","reported_seq":14669,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:28.410642+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:28.410642+0000","last_peered":"2024-09-18T12:01:28.410642+0000","last_clean":"2024-09-18T12:01:28.410642+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:28.410642+0000","last_undegraded":"2024-09-18T12:01:28.410642+0000","last_fullsized":"2024-09-18T12:01:28.410642+0000","mapping_epoch":29,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20164608,"num_objects":4923,"num_object_clones":0,"num_object_copies":9846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11701,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14572","reported_seq":14760,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.506771+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:28.506771+0000","last_peered":"2024-09-18T12:01:28.506771+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:28.506771+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20418584,"num_objects":4986,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11430,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14390","reported_seq":14421,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:28.443439+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:28.443439+0000","last_peered":"2024-09-18T12:01:28.443439+0000","last_clean":"2024-09-18T12:01:28.443439+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:28.443439+0000","last_undegraded":"2024-09-18T12:01:28.443439+0000","last_fullsized":"2024-09-18T12:01:28.443439+0000","mapping_epoch":29,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19947520,"num_objects":4870,"num_object_clones":0,"num_object_copies":9740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14390,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14455","reported_seq":14648,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:28.294917+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:28.294917+0000","last_peered":"2024-09-18T12:01:28.294917+0000","last_clean":"2024-09-18T12:01:28.294917+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:28.294917+0000","last_undegraded":"2024-09-18T12:01:28.294917+0000","last_fullsized":"2024-09-18T12:01:28.294917+0000","mapping_epoch":29,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20447232,"num_objects":4992,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11620,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14427","reported_seq":14510,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.493148+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:28.493148+0000","last_peered":"2024-09-18T12:01:28.493148+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:28.493148+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20464850,"num_objects":4998,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14425,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14455","reported_seq":14544,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.458250+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:28.458250+0000","last_peered":"2024-09-18T12:01:28.458250+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:28.458250+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141266,"num_objects":4919,"num_object_clones":0,"num_object_copies":9838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14453,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1360}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14709","reported_seq":14936,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:28.506961+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:28.506961+0000","last_peered":"2024-09-18T12:01:28.506961+0000","last_clean":"2024-09-18T12:01:28.506961+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:28.506961+0000","last_undegraded":"2024-09-18T12:01:28.506961+0000","last_fullsized":"2024-09-18T12:01:28.506961+0000","mapping_epoch":29,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20041728,"num_objects":4893,"num_object_clones":0,"num_object_copies":9786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11026,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14442","reported_seq":21572,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:29.541850+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:29.541850+0000","last_peered":"2024-09-18T12:01:29.541850+0000","last_clean":"2024-09-18T12:01:29.541850+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:29.541850+0000","last_undegraded":"2024-09-18T12:01:29.541850+0000","last_fullsized":"2024-09-18T12:01:29.541850+0000","mapping_epoch":38,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20329682,"num_objects":4965,"num_object_clones":0,"num_object_copies":9930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14440,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14481","reported_seq":14561,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.445640+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:28.445640+0000","last_peered":"2024-09-18T12:01:28.445640+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:28.445640+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337874,"num_objects":4967,"num_object_clones":0,"num_object_copies":9934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14479,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1435}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14530","reported_seq":14588,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.451594+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:28.451594+0000","last_peered":"2024-09-18T12:01:28.451594+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:28.451594+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20206802,"num_objects":4935,"num_object_clones":0,"num_object_copies":9870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14528,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14550","reported_seq":20982,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.507720+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:28.507720+0000","last_peered":"2024-09-18T12:01:28.507720+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:28.507720+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20267008,"num_objects":4948,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":485,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11030,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3110,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4463}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14549","reported_seq":14631,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.481869+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:28.481869+0000","last_peered":"2024-09-18T12:01:28.481869+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:28.481869+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20394601,"num_objects":4980,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14548,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1445}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14615","reported_seq":21763,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:29.457628+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:29.457628+0000","last_peered":"2024-09-18T12:01:29.457628+0000","last_clean":"2024-09-18T12:01:29.457628+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:29.457628+0000","last_undegraded":"2024-09-18T12:01:29.457628+0000","last_fullsized":"2024-09-18T12:01:29.457628+0000","mapping_epoch":40,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20100923,"num_objects":4910,"num_object_clones":0,"num_object_copies":9820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14612,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14376","reported_seq":14459,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.504487+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:28.504487+0000","last_peered":"2024-09-18T12:01:28.504487+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:28.504487+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4300","ondisk_log_start":"37'4300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19752146,"num_objects":4824,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14374,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14562","reported_seq":14649,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:28.460823+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:28.460823+0000","last_peered":"2024-09-18T12:01:28.460823+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:28.460823+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20277668,"num_objects":4954,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14559,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":323366988,"num_objects":78965,"num_object_clones":0,"num_object_copies":157930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78965,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":216250,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10138,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":820117504,"data_stored":820046738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160757,"ondisk_log_size":160757,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051675,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578432,"kb_used_data":272248,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93793408,"statfs":{"total":96636764160,"available":96044449792,"internally_reserved":0,"allocated":278781952,"data_stored":278543836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.24199999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.26700000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.434}]}]},{"osd":1,"up_from":35,"seq":150323855438,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542768,"kb_used_data":236584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93829072,"statfs":{"total":96636764160,"available":96080969728,"internally_reserved":0,"allocated":242262016,"data_stored":241968093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":1.766}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":1.444},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":1.8009999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599816,"kb_used_data":293696,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93772024,"statfs":{"total":96636764160,"available":96022552576,"internally_reserved":0,"allocated":300744704,"data_stored":300507038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":1.74},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":2.294}]},{"osd":2,"last update":"Wed Sep 18 12:00:30 2024","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":1.5189999999999999,"15min":0.81899999999999995},"min":{"1min":0.20200000000000001,"5min":0.185,"15min":0.16500000000000001},"max":{"1min":8.2170000000000005,"5min":8.6080000000000005,"15min":8.6080000000000005},"last":2.5129999999999999},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.3600000000000001,"15min":0.66800000000000004},"min":{"1min":0.252,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":9.4700000000000006,"5min":9.7970000000000006,"15min":9.7970000000000006},"last":2.4609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":300240896,"data_stored":300237417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":241651712,"data_stored":241591976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278224896,"data_stored":278217345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:35.394 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:36.174 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:36.194 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:36.203 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-18T12:01:35.203230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":314699871,"num_objects":76851,"num_object_clones":0,"num_object_copies":153702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76851,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":218368,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10305,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160976,"ondisk_log_size":160976,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1703764,"kb_used_data":785276,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281411756,"statfs":{"total":289910292480,"available":288165638144,"internally_reserved":0,"allocated":804122624,"data_stored":803352919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,3,8,8,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26497024,"num_objects":-6469,"num_object_clones":0,"num_object_copies":-12938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6469,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":504,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002756"},"pg_stats":[{"pgid":"2.f","version":"41'14765","reported_seq":14838,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.460431+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:33.460431+0000","last_peered":"2024-09-18T12:01:33.460431+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:33.460431+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19542016,"num_objects":4771,"num_object_clones":0,"num_object_copies":9542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14765,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1172}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14652","reported_seq":14812,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:33.521490+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:33.521490+0000","last_peered":"2024-09-18T12:01:33.521490+0000","last_clean":"2024-09-18T12:01:33.521490+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:33.521490+0000","last_undegraded":"2024-09-18T12:01:33.521490+0000","last_fullsized":"2024-09-18T12:01:33.521490+0000","mapping_epoch":29,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19578880,"num_objects":4780,"num_object_clones":0,"num_object_copies":9560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14704","reported_seq":14892,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.368578+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:33.368578+0000","last_peered":"2024-09-18T12:01:33.368578+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:33.368578+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19877912,"num_objects":4854,"num_object_clones":0,"num_object_copies":9708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11562,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1318}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14527","reported_seq":14558,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:33.438114+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:33.438114+0000","last_peered":"2024-09-18T12:01:33.438114+0000","last_clean":"2024-09-18T12:01:33.438114+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:33.438114+0000","last_undegraded":"2024-09-18T12:01:33.438114+0000","last_fullsized":"2024-09-18T12:01:33.438114+0000","mapping_epoch":29,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19386368,"num_objects":4733,"num_object_clones":0,"num_object_copies":9466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14527,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14591","reported_seq":14784,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:33.521763+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:33.521763+0000","last_peered":"2024-09-18T12:01:33.521763+0000","last_clean":"2024-09-18T12:01:33.521763+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:33.521763+0000","last_undegraded":"2024-09-18T12:01:33.521763+0000","last_fullsized":"2024-09-18T12:01:33.521763+0000","mapping_epoch":29,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19890176,"num_objects":4856,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11756,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14552","reported_seq":14635,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.485831+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:33.485831+0000","last_peered":"2024-09-18T12:01:33.485831+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:33.485831+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19952850,"num_objects":4873,"num_object_clones":0,"num_object_copies":9746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14550,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14579","reported_seq":14668,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.414120+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:33.414120+0000","last_peered":"2024-09-18T12:01:33.414120+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:33.414120+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19633362,"num_objects":4795,"num_object_clones":0,"num_object_copies":9590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14577,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14850","reported_seq":15077,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:33.433755+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:33.433755+0000","last_peered":"2024-09-18T12:01:33.433755+0000","last_clean":"2024-09-18T12:01:33.433755+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:33.433755+0000","last_undegraded":"2024-09-18T12:01:33.433755+0000","last_fullsized":"2024-09-18T12:01:33.433755+0000","mapping_epoch":29,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19464192,"num_objects":4752,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11167,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14578","reported_seq":21708,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:34.545458+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:34.545458+0000","last_peered":"2024-09-18T12:01:34.545458+0000","last_clean":"2024-09-18T12:01:34.545458+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:34.545458+0000","last_undegraded":"2024-09-18T12:01:34.545458+0000","last_fullsized":"2024-09-18T12:01:34.545458+0000","mapping_epoch":38,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19772626,"num_objects":4829,"num_object_clones":0,"num_object_copies":9658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14576,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14613","reported_seq":14693,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.320274+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:33.320274+0000","last_peered":"2024-09-18T12:01:33.320274+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:33.320274+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19797202,"num_objects":4835,"num_object_clones":0,"num_object_copies":9670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14611,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1303}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14679","reported_seq":14737,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.397640+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:33.397640+0000","last_peered":"2024-09-18T12:01:33.397640+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:33.397640+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19596498,"num_objects":4786,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14677,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1236}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14669","reported_seq":21433,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.468242+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:33.468242+0000","last_peered":"2024-09-18T12:01:33.468242+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:33.468242+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19779584,"num_objects":4829,"num_object_clones":0,"num_object_copies":9658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11149,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3276,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14673","reported_seq":14756,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.486645+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:33.486645+0000","last_peered":"2024-09-18T12:01:33.486645+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:33.486645+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19886697,"num_objects":4856,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14672,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1321}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14743","reported_seq":21891,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:34.521083+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:34.521083+0000","last_peered":"2024-09-18T12:01:34.521083+0000","last_clean":"2024-09-18T12:01:34.521083+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:34.521083+0000","last_undegraded":"2024-09-18T12:01:34.521083+0000","last_fullsized":"2024-09-18T12:01:34.521083+0000","mapping_epoch":40,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19576635,"num_objects":4782,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14740,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14500","reported_seq":14583,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.467521+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:33.467521+0000","last_peered":"2024-09-18T12:01:33.467521+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:33.467521+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19244242,"num_objects":4700,"num_object_clones":0,"num_object_copies":9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14498,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14698","reported_seq":14785,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.415671+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:33.415671+0000","last_peered":"2024-09-18T12:01:33.415671+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:33.415671+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19720612,"num_objects":4818,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14695,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1295}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":314699852,"num_objects":76849,"num_object_clones":0,"num_object_copies":153698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":218366,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10304,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802578432,"data_stored":802507666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160973,"ondisk_log_size":160973,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051676,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578096,"kb_used_data":267752,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93793744,"statfs":{"total":96636764160,"available":96044793856,"internally_reserved":0,"allocated":274178048,"data_stored":273939932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.24099999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":1.286}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":5.3630000000000004},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":3.8050000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855439,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528136,"kb_used_data":230272,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93843704,"statfs":{"total":96636764160,"available":96095952896,"internally_reserved":0,"allocated":235798528,"data_stored":235504605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,0,4,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.28699999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597532,"kb_used_data":287252,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93774308,"statfs":{"total":96636764160,"available":96024891392,"internally_reserved":0,"allocated":294146048,"data_stored":293908382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":2.3290000000000002},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":2.161}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":2.516},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293703680,"data_stored":293700201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235163648,"data_stored":235103912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":273711104,"data_stored":273703553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:36.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:36.881 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:36.908 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:36.929 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-18T12:01:35.203230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":314699871,"num_objects":76851,"num_object_clones":0,"num_object_copies":153702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76851,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":218368,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10305,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160976,"ondisk_log_size":160976,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1703764,"kb_used_data":785276,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281411756,"statfs":{"total":289910292480,"available":288165638144,"internally_reserved":0,"allocated":804122624,"data_stored":803352919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,3,8,8,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26497024,"num_objects":-6469,"num_object_clones":0,"num_object_copies":-12938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6469,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":504,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002756"},"pg_stats":[{"pgid":"2.f","version":"41'14765","reported_seq":14838,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.460431+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:33.460431+0000","last_peered":"2024-09-18T12:01:33.460431+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:33.460431+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19542016,"num_objects":4771,"num_object_clones":0,"num_object_copies":9542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14765,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1172}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14652","reported_seq":14812,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:33.521490+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:33.521490+0000","last_peered":"2024-09-18T12:01:33.521490+0000","last_clean":"2024-09-18T12:01:33.521490+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:33.521490+0000","last_undegraded":"2024-09-18T12:01:33.521490+0000","last_fullsized":"2024-09-18T12:01:33.521490+0000","mapping_epoch":29,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19578880,"num_objects":4780,"num_object_clones":0,"num_object_copies":9560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14704","reported_seq":14892,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.368578+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:33.368578+0000","last_peered":"2024-09-18T12:01:33.368578+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:33.368578+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19877912,"num_objects":4854,"num_object_clones":0,"num_object_copies":9708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11562,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1318}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14527","reported_seq":14558,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:33.438114+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:33.438114+0000","last_peered":"2024-09-18T12:01:33.438114+0000","last_clean":"2024-09-18T12:01:33.438114+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:33.438114+0000","last_undegraded":"2024-09-18T12:01:33.438114+0000","last_fullsized":"2024-09-18T12:01:33.438114+0000","mapping_epoch":29,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19386368,"num_objects":4733,"num_object_clones":0,"num_object_copies":9466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14527,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14591","reported_seq":14784,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:33.521763+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:33.521763+0000","last_peered":"2024-09-18T12:01:33.521763+0000","last_clean":"2024-09-18T12:01:33.521763+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:33.521763+0000","last_undegraded":"2024-09-18T12:01:33.521763+0000","last_fullsized":"2024-09-18T12:01:33.521763+0000","mapping_epoch":29,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19890176,"num_objects":4856,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11756,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14552","reported_seq":14635,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.485831+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:33.485831+0000","last_peered":"2024-09-18T12:01:33.485831+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:33.485831+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19952850,"num_objects":4873,"num_object_clones":0,"num_object_copies":9746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14550,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14579","reported_seq":14668,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.414120+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:33.414120+0000","last_peered":"2024-09-18T12:01:33.414120+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:33.414120+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19633362,"num_objects":4795,"num_object_clones":0,"num_object_copies":9590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14577,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14850","reported_seq":15077,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:33.433755+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:33.433755+0000","last_peered":"2024-09-18T12:01:33.433755+0000","last_clean":"2024-09-18T12:01:33.433755+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:33.433755+0000","last_undegraded":"2024-09-18T12:01:33.433755+0000","last_fullsized":"2024-09-18T12:01:33.433755+0000","mapping_epoch":29,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19464192,"num_objects":4752,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11167,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14578","reported_seq":21708,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:34.545458+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:34.545458+0000","last_peered":"2024-09-18T12:01:34.545458+0000","last_clean":"2024-09-18T12:01:34.545458+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:34.545458+0000","last_undegraded":"2024-09-18T12:01:34.545458+0000","last_fullsized":"2024-09-18T12:01:34.545458+0000","mapping_epoch":38,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19772626,"num_objects":4829,"num_object_clones":0,"num_object_copies":9658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14576,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14613","reported_seq":14693,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.320274+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:33.320274+0000","last_peered":"2024-09-18T12:01:33.320274+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:33.320274+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19797202,"num_objects":4835,"num_object_clones":0,"num_object_copies":9670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14611,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1303}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14679","reported_seq":14737,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.397640+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:33.397640+0000","last_peered":"2024-09-18T12:01:33.397640+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:33.397640+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19596498,"num_objects":4786,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14677,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1236}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14669","reported_seq":21433,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.468242+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:33.468242+0000","last_peered":"2024-09-18T12:01:33.468242+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:33.468242+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19779584,"num_objects":4829,"num_object_clones":0,"num_object_copies":9658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11149,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3276,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14673","reported_seq":14756,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.486645+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:33.486645+0000","last_peered":"2024-09-18T12:01:33.486645+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:33.486645+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19886697,"num_objects":4856,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14672,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1321}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14743","reported_seq":21891,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:34.521083+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:34.521083+0000","last_peered":"2024-09-18T12:01:34.521083+0000","last_clean":"2024-09-18T12:01:34.521083+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:34.521083+0000","last_undegraded":"2024-09-18T12:01:34.521083+0000","last_fullsized":"2024-09-18T12:01:34.521083+0000","mapping_epoch":40,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19576635,"num_objects":4782,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14740,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14500","reported_seq":14583,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.467521+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:33.467521+0000","last_peered":"2024-09-18T12:01:33.467521+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:33.467521+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19244242,"num_objects":4700,"num_object_clones":0,"num_object_copies":9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14498,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14698","reported_seq":14785,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.415671+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:33.415671+0000","last_peered":"2024-09-18T12:01:33.415671+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:33.415671+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19720612,"num_objects":4818,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14695,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1295}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":314699852,"num_objects":76849,"num_object_clones":0,"num_object_copies":153698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":218366,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10304,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802578432,"data_stored":802507666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160973,"ondisk_log_size":160973,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051676,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578096,"kb_used_data":267752,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93793744,"statfs":{"total":96636764160,"available":96044793856,"internally_reserved":0,"allocated":274178048,"data_stored":273939932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.24099999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":1.286}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":5.3630000000000004},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":3.8050000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855439,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528136,"kb_used_data":230272,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93843704,"statfs":{"total":96636764160,"available":96095952896,"internally_reserved":0,"allocated":235798528,"data_stored":235504605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,0,4,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.28699999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597532,"kb_used_data":287252,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93774308,"statfs":{"total":96636764160,"available":96024891392,"internally_reserved":0,"allocated":294146048,"data_stored":293908382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":2.3290000000000002},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":2.161}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":2.516},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293703680,"data_stored":293700201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235163648,"data_stored":235103912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":273711104,"data_stored":273703553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:36.930 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:01:36.930 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'14765", 'reported_seq': 14838, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:33.460431+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:01:33.460431+0000', 'last_peered': '2024-09-18T12:01:33.460431+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:01:33.460431+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'4700", 'ondisk_log_start': "37'4700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19542016, 'num_objects': 4771, 'num_object_clones': 0, 'num_object_copies': 9542, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14765, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1172}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:36.930 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:01:36.930 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'14704", 'reported_seq': 14892, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:33.368578+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:01:33.368578+0000', 'last_peered': '2024-09-18T12:01:33.368578+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:01:33.368578+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'4700", 'ondisk_log_start': "37'4700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19877912, 'num_objects': 4854, 'num_object_clones': 0, 'num_object_copies': 9708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4854, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11562, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:36.931 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:01:36.931 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'14552", 'reported_seq': 14635, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:33.485831+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:01:33.485831+0000', 'last_peered': '2024-09-18T12:01:33.485831+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:01:33.485831+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'4500", 'ondisk_log_start': "37'4500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19952850, 'num_objects': 4873, 'num_object_clones': 0, 'num_object_copies': 9746, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14550, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1346}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:36.931 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:01:36.931 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'14579", 'reported_seq': 14668, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:33.414120+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:01:33.414120+0000', 'last_peered': '2024-09-18T12:01:33.414120+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:01:33.414120+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'4500", 'ondisk_log_start': "37'4500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19633362, 'num_objects': 4795, 'num_object_clones': 0, 'num_object_copies': 9590, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4795, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14577, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1236}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:36.931 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:01:36.931 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'14613", 'reported_seq': 14693, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:33.320274+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:01:33.320274+0000', 'last_peered': '2024-09-18T12:01:33.320274+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:01:33.320274+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'4600", 'ondisk_log_start': "37'4600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19797202, 'num_objects': 4835, 'num_object_clones': 0, 'num_object_copies': 9670, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4835, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14611, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1303}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:36.931 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:01:36.931 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'14679", 'reported_seq': 14737, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:33.397640+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:01:33.397640+0000', 'last_peered': '2024-09-18T12:01:33.397640+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:01:33.397640+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'4600", 'ondisk_log_start': "37'4600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19596498, 'num_objects': 4786, 'num_object_clones': 0, 'num_object_copies': 9572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14677, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1236}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:36.931 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:01:36.931 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'14669", 'reported_seq': 21433, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:33.468242+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:01:33.468242+0000', 'last_peered': '2024-09-18T12:01:33.468242+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:01:33.468242+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "37'4600", 'ondisk_log_start': "37'4600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19779584, 'num_objects': 4829, 'num_object_clones': 0, 'num_object_copies': 9658, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11149, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3276, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 4510}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:36.931 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:01:36.932 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'14673", 'reported_seq': 14756, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:33.486645+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:01:33.486645+0000', 'last_peered': '2024-09-18T12:01:33.486645+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:01:33.486645+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'4600", 'ondisk_log_start': "37'4600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19886697, 'num_objects': 4856, 'num_object_clones': 0, 'num_object_copies': 9712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14672, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1321}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:36.932 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:01:36.932 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'14500", 'reported_seq': 14583, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:33.467521+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:01:33.467521+0000', 'last_peered': '2024-09-18T12:01:33.467521+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:01:33.467521+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'4400", 'ondisk_log_start': "37'4400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19244242, 'num_objects': 4700, 'num_object_clones': 0, 'num_object_copies': 9400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4700, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14498, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1107}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:36.932 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:01:36.932 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'14698", 'reported_seq': 14785, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:33.415671+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:01:33.415671+0000', 'last_peered': '2024-09-18T12:01:33.415671+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:01:33.415671+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'4600", 'ondisk_log_start': "37'4600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19720612, 'num_objects': 4818, 'num_object_clones': 0, 'num_object_copies': 9636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14695, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1295}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:36.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:01:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:37.620 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32288/153702 objects degraded (21.007%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":497,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":76851,"data_bytes":314699871,"bytes_used":1744654336,"bytes_avail":288165638144,"bytes_total":289910292480,"degraded_objects":32288,"degraded_total":153702,"degraded_ratio":0.21006883449792454,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":338,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":358},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:01:37.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:01:37.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:38.344 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:38.363 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-18T12:01:37.203715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":314699871,"num_objects":76851,"num_object_clones":0,"num_object_copies":153702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76851,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":218368,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10305,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160976,"ondisk_log_size":160976,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1703764,"kb_used_data":785276,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281411756,"statfs":{"total":289910292480,"available":288165638144,"internally_reserved":0,"allocated":804122624,"data_stored":803352919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,3,8,8,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17620992,"num_objects":-4302,"num_object_clones":0,"num_object_copies":-8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002818"},"pg_stats":[{"pgid":"2.f","version":"41'14765","reported_seq":14838,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.460431+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:33.460431+0000","last_peered":"2024-09-18T12:01:33.460431+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:33.460431+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19542016,"num_objects":4771,"num_object_clones":0,"num_object_copies":9542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14765,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1172}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14652","reported_seq":14812,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:33.521490+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:33.521490+0000","last_peered":"2024-09-18T12:01:33.521490+0000","last_clean":"2024-09-18T12:01:33.521490+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:33.521490+0000","last_undegraded":"2024-09-18T12:01:33.521490+0000","last_fullsized":"2024-09-18T12:01:33.521490+0000","mapping_epoch":29,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19578880,"num_objects":4780,"num_object_clones":0,"num_object_copies":9560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14704","reported_seq":14892,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.368578+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:33.368578+0000","last_peered":"2024-09-18T12:01:33.368578+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:33.368578+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19877912,"num_objects":4854,"num_object_clones":0,"num_object_copies":9708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11562,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1318}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14527","reported_seq":14558,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:33.438114+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:33.438114+0000","last_peered":"2024-09-18T12:01:33.438114+0000","last_clean":"2024-09-18T12:01:33.438114+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:33.438114+0000","last_undegraded":"2024-09-18T12:01:33.438114+0000","last_fullsized":"2024-09-18T12:01:33.438114+0000","mapping_epoch":29,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19386368,"num_objects":4733,"num_object_clones":0,"num_object_copies":9466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14527,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14591","reported_seq":14784,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:33.521763+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:33.521763+0000","last_peered":"2024-09-18T12:01:33.521763+0000","last_clean":"2024-09-18T12:01:33.521763+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:33.521763+0000","last_undegraded":"2024-09-18T12:01:33.521763+0000","last_fullsized":"2024-09-18T12:01:33.521763+0000","mapping_epoch":29,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19890176,"num_objects":4856,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11756,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14552","reported_seq":14635,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.485831+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:33.485831+0000","last_peered":"2024-09-18T12:01:33.485831+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:33.485831+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19952850,"num_objects":4873,"num_object_clones":0,"num_object_copies":9746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14550,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14579","reported_seq":14668,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.414120+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:33.414120+0000","last_peered":"2024-09-18T12:01:33.414120+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:33.414120+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19633362,"num_objects":4795,"num_object_clones":0,"num_object_copies":9590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14577,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14850","reported_seq":15077,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:33.433755+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:33.433755+0000","last_peered":"2024-09-18T12:01:33.433755+0000","last_clean":"2024-09-18T12:01:33.433755+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:33.433755+0000","last_undegraded":"2024-09-18T12:01:33.433755+0000","last_fullsized":"2024-09-18T12:01:33.433755+0000","mapping_epoch":29,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19464192,"num_objects":4752,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11167,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14578","reported_seq":21708,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:34.545458+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:34.545458+0000","last_peered":"2024-09-18T12:01:34.545458+0000","last_clean":"2024-09-18T12:01:34.545458+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:34.545458+0000","last_undegraded":"2024-09-18T12:01:34.545458+0000","last_fullsized":"2024-09-18T12:01:34.545458+0000","mapping_epoch":38,"log_start":"37'4500","ondisk_log_start":"37'4500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19772626,"num_objects":4829,"num_object_clones":0,"num_object_copies":9658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14576,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14613","reported_seq":14693,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.320274+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:33.320274+0000","last_peered":"2024-09-18T12:01:33.320274+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:33.320274+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19797202,"num_objects":4835,"num_object_clones":0,"num_object_copies":9670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14611,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1303}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14679","reported_seq":14737,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.397640+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:33.397640+0000","last_peered":"2024-09-18T12:01:33.397640+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:33.397640+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19596498,"num_objects":4786,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14677,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1236}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14669","reported_seq":21433,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.468242+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:33.468242+0000","last_peered":"2024-09-18T12:01:33.468242+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:33.468242+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19779584,"num_objects":4829,"num_object_clones":0,"num_object_copies":9658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11149,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3276,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14673","reported_seq":14756,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.486645+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:33.486645+0000","last_peered":"2024-09-18T12:01:33.486645+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:33.486645+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19886697,"num_objects":4856,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14672,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1321}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14743","reported_seq":21891,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:34.521083+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:34.521083+0000","last_peered":"2024-09-18T12:01:34.521083+0000","last_clean":"2024-09-18T12:01:34.521083+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:34.521083+0000","last_undegraded":"2024-09-18T12:01:34.521083+0000","last_fullsized":"2024-09-18T12:01:34.521083+0000","mapping_epoch":40,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19576635,"num_objects":4782,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14740,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14500","reported_seq":14583,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.467521+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:33.467521+0000","last_peered":"2024-09-18T12:01:33.467521+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:33.467521+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4400","ondisk_log_start":"37'4400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19244242,"num_objects":4700,"num_object_clones":0,"num_object_copies":9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14498,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14698","reported_seq":14785,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:33.415671+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:33.415671+0000","last_peered":"2024-09-18T12:01:33.415671+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:33.415671+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19720612,"num_objects":4818,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14695,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1295}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":314699852,"num_objects":76849,"num_object_clones":0,"num_object_copies":153698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":218366,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10304,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802578432,"data_stored":802507666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160973,"ondisk_log_size":160973,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051676,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578096,"kb_used_data":267752,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93793744,"statfs":{"total":96636764160,"available":96044793856,"internally_reserved":0,"allocated":274178048,"data_stored":273939932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.24099999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":1.286}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":5.3630000000000004},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":3.8050000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855439,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528136,"kb_used_data":230272,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93843704,"statfs":{"total":96636764160,"available":96095952896,"internally_reserved":0,"allocated":235798528,"data_stored":235504605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,0,4,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.28699999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597532,"kb_used_data":287252,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93774308,"statfs":{"total":96636764160,"available":96024891392,"internally_reserved":0,"allocated":294146048,"data_stored":293908382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":2.3290000000000002},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":2.161}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":2.516},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293703680,"data_stored":293700201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":235163648,"data_stored":235103912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":273711104,"data_stored":273703553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:41.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:41.945 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:41.975 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:42.002 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-18T12:01:41.204625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":305709151,"num_objects":74656,"num_object_clones":0,"num_object_copies":149312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74656,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220563,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10460,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160771,"ondisk_log_size":160771,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1670560,"kb_used_data":768712,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281444960,"statfs":{"total":289910292480,"available":288199639040,"internally_reserved":0,"allocated":787161088,"data_stored":786391383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,6,10,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18800640,"num_objects":-4590,"num_object_clones":0,"num_object_copies":-9180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4590,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002813"},"pg_stats":[{"pgid":"2.f","version":"41'14898","reported_seq":14971,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.530544+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:38.530544+0000","last_peered":"2024-09-18T12:01:38.530544+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:38.530544+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18997248,"num_objects":4638,"num_object_clones":0,"num_object_copies":9276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14898,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1039}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14783","reported_seq":14943,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:38.488966+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:38.488966+0000","last_peered":"2024-09-18T12:01:38.488966+0000","last_clean":"2024-09-18T12:01:38.488966+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:38.488966+0000","last_undegraded":"2024-09-18T12:01:38.488966+0000","last_fullsized":"2024-09-18T12:01:38.488966+0000","mapping_epoch":29,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19042304,"num_objects":4649,"num_object_clones":0,"num_object_copies":9298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14846","reported_seq":15033,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.528666+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:38.528666+0000","last_peered":"2024-09-18T12:01:38.528666+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:38.528666+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19296280,"num_objects":4712,"num_object_clones":0,"num_object_copies":9424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4712,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11704,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1176}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14653","reported_seq":14684,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:38.500602+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:38.500602+0000","last_peered":"2024-09-18T12:01:38.500602+0000","last_clean":"2024-09-18T12:01:38.500602+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:38.500602+0000","last_undegraded":"2024-09-18T12:01:38.500602+0000","last_fullsized":"2024-09-18T12:01:38.500602+0000","mapping_epoch":29,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18870272,"num_objects":4607,"num_object_clones":0,"num_object_copies":9214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14653,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14739","reported_seq":14932,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:38.502828+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:38.502828+0000","last_peered":"2024-09-18T12:01:38.502828+0000","last_clean":"2024-09-18T12:01:38.502828+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:38.502828+0000","last_undegraded":"2024-09-18T12:01:38.502828+0000","last_fullsized":"2024-09-18T12:01:38.502828+0000","mapping_epoch":29,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19283968,"num_objects":4708,"num_object_clones":0,"num_object_copies":9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11904,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14689","reported_seq":14772,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.560949+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:38.560949+0000","last_peered":"2024-09-18T12:01:38.560949+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:38.560949+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19391698,"num_objects":4736,"num_object_clones":0,"num_object_copies":9472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14687,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14718","reported_seq":14807,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.559598+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:38.559598+0000","last_peered":"2024-09-18T12:01:38.559598+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:38.559598+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19064018,"num_objects":4656,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14716,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1097}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14973","reported_seq":15200,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:38.437733+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:38.437733+0000","last_peered":"2024-09-18T12:01:38.437733+0000","last_clean":"2024-09-18T12:01:38.437733+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:38.437733+0000","last_undegraded":"2024-09-18T12:01:38.437733+0000","last_fullsized":"2024-09-18T12:01:38.437733+0000","mapping_epoch":29,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18960384,"num_objects":4629,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11290,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14722","reported_seq":21852,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:39.563597+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:39.563597+0000","last_peered":"2024-09-18T12:01:39.563597+0000","last_clean":"2024-09-18T12:01:39.563597+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:39.563597+0000","last_undegraded":"2024-09-18T12:01:39.563597+0000","last_fullsized":"2024-09-18T12:01:39.563597+0000","mapping_epoch":38,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19182802,"num_objects":4685,"num_object_clones":0,"num_object_copies":9370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14720,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14754","reported_seq":14832,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.561987+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:38.561987+0000","last_peered":"2024-09-18T12:01:38.561987+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:38.561987+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19219666,"num_objects":4694,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14752,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1162}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14802","reported_seq":14860,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.500604+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:38.500604+0000","last_peered":"2024-09-18T12:01:38.500604+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:38.500604+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19092690,"num_objects":4663,"num_object_clones":0,"num_object_copies":9326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14800,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1113}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14833","reported_seq":21907,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.536981+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:38.536981+0000","last_peered":"2024-09-18T12:01:38.536981+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:38.536981+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19107840,"num_objects":4665,"num_object_clones":0,"num_object_copies":9330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11313,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3431,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14801","reported_seq":14885,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.424432+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:38.424432+0000","last_peered":"2024-09-18T12:01:38.424432+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:38.424432+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19362409,"num_objects":4728,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14800,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1193}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14889","reported_seq":22037,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:39.594940+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:39.594940+0000","last_peered":"2024-09-18T12:01:39.594940+0000","last_clean":"2024-09-18T12:01:39.594940+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:39.594940+0000","last_undegraded":"2024-09-18T12:01:39.594940+0000","last_fullsized":"2024-09-18T12:01:39.594940+0000","mapping_epoch":40,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18978619,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14886,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14630","reported_seq":14713,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.383845+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:38.383845+0000","last_peered":"2024-09-18T12:01:38.383845+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:38.383845+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18711762,"num_objects":4570,"num_object_clones":0,"num_object_copies":9140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14628,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":977}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14838","reported_seq":14925,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.556756+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:38.556756+0000","last_peered":"2024-09-18T12:01:38.556756+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:38.556756+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19147172,"num_objects":4678,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14835,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":305709132,"num_objects":74654,"num_object_clones":0,"num_object_copies":149308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74654,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220561,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10459,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":784752640,"data_stored":784681874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160768,"ondisk_log_size":160768,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051677,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562020,"kb_used_data":264156,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93809820,"statfs":{"total":96636764160,"available":96061255680,"internally_reserved":0,"allocated":270495744,"data_stored":270257628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.436}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.32600000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855440,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525680,"kb_used_data":223656,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93846160,"statfs":{"total":96636764160,"available":96098467840,"internally_reserved":0,"allocated":229023744,"data_stored":228729821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.214},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.34899999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.379},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":6.4900000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582860,"kb_used_data":280900,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93788980,"statfs":{"total":96636764160,"available":96039915520,"internally_reserved":0,"allocated":287641600,"data_stored":287403934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":2.3290000000000002},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":2.161}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":2.516},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":287125504,"data_stored":287122025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228372480,"data_stored":228312744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":269254656,"data_stored":269247105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:42.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:42.729 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:42.734 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:42.769 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-18T12:01:41.204625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":305709151,"num_objects":74656,"num_object_clones":0,"num_object_copies":149312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74656,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220563,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10460,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160771,"ondisk_log_size":160771,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1670560,"kb_used_data":768712,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281444960,"statfs":{"total":289910292480,"available":288199639040,"internally_reserved":0,"allocated":787161088,"data_stored":786391383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,6,10,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18800640,"num_objects":-4590,"num_object_clones":0,"num_object_copies":-9180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4590,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002813"},"pg_stats":[{"pgid":"2.f","version":"41'14898","reported_seq":14971,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.530544+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:38.530544+0000","last_peered":"2024-09-18T12:01:38.530544+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:38.530544+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18997248,"num_objects":4638,"num_object_clones":0,"num_object_copies":9276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14898,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1039}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14783","reported_seq":14943,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:38.488966+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:38.488966+0000","last_peered":"2024-09-18T12:01:38.488966+0000","last_clean":"2024-09-18T12:01:38.488966+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:38.488966+0000","last_undegraded":"2024-09-18T12:01:38.488966+0000","last_fullsized":"2024-09-18T12:01:38.488966+0000","mapping_epoch":29,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19042304,"num_objects":4649,"num_object_clones":0,"num_object_copies":9298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14846","reported_seq":15033,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.528666+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:38.528666+0000","last_peered":"2024-09-18T12:01:38.528666+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:38.528666+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19296280,"num_objects":4712,"num_object_clones":0,"num_object_copies":9424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4712,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11704,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1176}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14653","reported_seq":14684,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:38.500602+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:38.500602+0000","last_peered":"2024-09-18T12:01:38.500602+0000","last_clean":"2024-09-18T12:01:38.500602+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:38.500602+0000","last_undegraded":"2024-09-18T12:01:38.500602+0000","last_fullsized":"2024-09-18T12:01:38.500602+0000","mapping_epoch":29,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18870272,"num_objects":4607,"num_object_clones":0,"num_object_copies":9214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14653,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14739","reported_seq":14932,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:38.502828+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:38.502828+0000","last_peered":"2024-09-18T12:01:38.502828+0000","last_clean":"2024-09-18T12:01:38.502828+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:38.502828+0000","last_undegraded":"2024-09-18T12:01:38.502828+0000","last_fullsized":"2024-09-18T12:01:38.502828+0000","mapping_epoch":29,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19283968,"num_objects":4708,"num_object_clones":0,"num_object_copies":9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11904,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14689","reported_seq":14772,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.560949+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:38.560949+0000","last_peered":"2024-09-18T12:01:38.560949+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:38.560949+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19391698,"num_objects":4736,"num_object_clones":0,"num_object_copies":9472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14687,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14718","reported_seq":14807,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.559598+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:38.559598+0000","last_peered":"2024-09-18T12:01:38.559598+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:38.559598+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19064018,"num_objects":4656,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14716,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1097}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14973","reported_seq":15200,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:38.437733+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:38.437733+0000","last_peered":"2024-09-18T12:01:38.437733+0000","last_clean":"2024-09-18T12:01:38.437733+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:38.437733+0000","last_undegraded":"2024-09-18T12:01:38.437733+0000","last_fullsized":"2024-09-18T12:01:38.437733+0000","mapping_epoch":29,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18960384,"num_objects":4629,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11290,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14722","reported_seq":21852,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:39.563597+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:39.563597+0000","last_peered":"2024-09-18T12:01:39.563597+0000","last_clean":"2024-09-18T12:01:39.563597+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:39.563597+0000","last_undegraded":"2024-09-18T12:01:39.563597+0000","last_fullsized":"2024-09-18T12:01:39.563597+0000","mapping_epoch":38,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19182802,"num_objects":4685,"num_object_clones":0,"num_object_copies":9370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14720,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14754","reported_seq":14832,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.561987+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:38.561987+0000","last_peered":"2024-09-18T12:01:38.561987+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:38.561987+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19219666,"num_objects":4694,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14752,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1162}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14802","reported_seq":14860,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.500604+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:38.500604+0000","last_peered":"2024-09-18T12:01:38.500604+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:38.500604+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19092690,"num_objects":4663,"num_object_clones":0,"num_object_copies":9326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14800,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1113}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14833","reported_seq":21907,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.536981+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:38.536981+0000","last_peered":"2024-09-18T12:01:38.536981+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:38.536981+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19107840,"num_objects":4665,"num_object_clones":0,"num_object_copies":9330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11313,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3431,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14801","reported_seq":14885,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.424432+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:38.424432+0000","last_peered":"2024-09-18T12:01:38.424432+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:38.424432+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19362409,"num_objects":4728,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14800,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1193}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14889","reported_seq":22037,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:39.594940+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:39.594940+0000","last_peered":"2024-09-18T12:01:39.594940+0000","last_clean":"2024-09-18T12:01:39.594940+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:39.594940+0000","last_undegraded":"2024-09-18T12:01:39.594940+0000","last_fullsized":"2024-09-18T12:01:39.594940+0000","mapping_epoch":40,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18978619,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14886,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14630","reported_seq":14713,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.383845+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:38.383845+0000","last_peered":"2024-09-18T12:01:38.383845+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:38.383845+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18711762,"num_objects":4570,"num_object_clones":0,"num_object_copies":9140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14628,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":977}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14838","reported_seq":14925,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.556756+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:38.556756+0000","last_peered":"2024-09-18T12:01:38.556756+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:38.556756+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19147172,"num_objects":4678,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14835,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":305709132,"num_objects":74654,"num_object_clones":0,"num_object_copies":149308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74654,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220561,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10459,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":784752640,"data_stored":784681874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160768,"ondisk_log_size":160768,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051677,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562020,"kb_used_data":264156,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93809820,"statfs":{"total":96636764160,"available":96061255680,"internally_reserved":0,"allocated":270495744,"data_stored":270257628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.436}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.32600000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855440,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525680,"kb_used_data":223656,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93846160,"statfs":{"total":96636764160,"available":96098467840,"internally_reserved":0,"allocated":229023744,"data_stored":228729821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.214},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.34899999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.379},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":6.4900000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582860,"kb_used_data":280900,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93788980,"statfs":{"total":96636764160,"available":96039915520,"internally_reserved":0,"allocated":287641600,"data_stored":287403934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":2.3290000000000002},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":2.161}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":2.516},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":287125504,"data_stored":287122025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228372480,"data_stored":228312744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":269254656,"data_stored":269247105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:42.770 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:01:42.770 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'14898", 'reported_seq': 14971, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:38.530544+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:01:38.530544+0000', 'last_peered': '2024-09-18T12:01:38.530544+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:01:38.530544+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'4800", 'ondisk_log_start': "37'4800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18997248, 'num_objects': 4638, 'num_object_clones': 0, 'num_object_copies': 9276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4638, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14898, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1039}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:42.770 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:01:42.770 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'14846", 'reported_seq': 15033, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:38.528666+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:01:38.528666+0000', 'last_peered': '2024-09-18T12:01:38.528666+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:01:38.528666+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'4800", 'ondisk_log_start': "37'4800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19296280, 'num_objects': 4712, 'num_object_clones': 0, 'num_object_copies': 9424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4712, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11704, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:42.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:01:42.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'14689", 'reported_seq': 14772, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:38.560949+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:01:38.560949+0000', 'last_peered': '2024-09-18T12:01:38.560949+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:01:38.560949+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'4600", 'ondisk_log_start': "37'4600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19391698, 'num_objects': 4736, 'num_object_clones': 0, 'num_object_copies': 9472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14687, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1209}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:42.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:01:42.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'14718", 'reported_seq': 14807, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:38.559598+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:01:38.559598+0000', 'last_peered': '2024-09-18T12:01:38.559598+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:01:38.559598+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'4700", 'ondisk_log_start': "37'4700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19064018, 'num_objects': 4656, 'num_object_clones': 0, 'num_object_copies': 9312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14716, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1097}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:42.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:01:42.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "41'14754", 'reported_seq': 14832, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:38.561987+0000', 'last_change': '2024-09-18T11:55:04.728869+0000', 'last_active': '2024-09-18T12:01:38.561987+0000', 'last_peered': '2024-09-18T12:01:38.561987+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:01:38.561987+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'4700", 'ondisk_log_start': "37'4700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19219666, 'num_objects': 4694, 'num_object_clones': 0, 'num_object_copies': 9388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3534, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14752, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1162}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:42.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:01:42.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'14802", 'reported_seq': 14860, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:38.500604+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:01:38.500604+0000', 'last_peered': '2024-09-18T12:01:38.500604+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:01:38.500604+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'4800", 'ondisk_log_start': "37'4800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19092690, 'num_objects': 4663, 'num_object_clones': 0, 'num_object_copies': 9326, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14800, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1113}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:42.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:01:42.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'14833", 'reported_seq': 21907, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:38.536981+0000', 'last_change': '2024-09-18T11:59:15.619746+0000', 'last_active': '2024-09-18T12:01:38.536981+0000', 'last_peered': '2024-09-18T12:01:38.536981+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:01:38.536981+0000', 'last_undegraded': '2024-09-18T11:55:04.308388+0000', 'last_fullsized': '2024-09-18T11:55:04.245085+0000', 'mapping_epoch': 36, 'log_start': "37'4800", 'ondisk_log_start': "37'4800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19107840, 'num_objects': 4665, 'num_object_clones': 0, 'num_object_copies': 9330, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11313, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3431, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 4501}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:42.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:01:42.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "41'14801", 'reported_seq': 14885, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:38.424432+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:01:38.424432+0000', 'last_peered': '2024-09-18T12:01:38.424432+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:01:38.424432+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'4800", 'ondisk_log_start': "37'4800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19362409, 'num_objects': 4728, 'num_object_clones': 0, 'num_object_copies': 9456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14800, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1193}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:42.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:01:42.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "41'14630", 'reported_seq': 14713, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:38.383845+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:01:38.383845+0000', 'last_peered': '2024-09-18T12:01:38.383845+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:01:38.383845+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'4600", 'ondisk_log_start': "37'4600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18711762, 'num_objects': 4570, 'num_object_clones': 0, 'num_object_copies': 9140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14628, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 977}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:42.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:01:42.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "41'14838", 'reported_seq': 14925, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:38.556756+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:01:38.556756+0000', 'last_peered': '2024-09-18T12:01:38.556756+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:01:38.556756+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'4800", 'ondisk_log_start': "37'4800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19147172, 'num_objects': 4678, 'num_object_clones': 0, 'num_object_copies': 9356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14835, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1155}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:42.772 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:01:43.583 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:43.583 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32133/149312 objects degraded (21.521%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":503,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":74656,"data_bytes":305709151,"bytes_used":1710653440,"bytes_avail":288199639040,"bytes_total":289910292480,"degraded_objects":32133,"degraded_total":149312,"degraded_ratio":0.21520708315473638,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":321,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":359},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:01:43.608 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:01:43.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:44.105 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:44.107 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:44.119 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-18T12:01:43.479235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":305709151,"num_objects":74656,"num_object_clones":0,"num_object_copies":149312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74656,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220563,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10460,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160771,"ondisk_log_size":160771,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1670560,"kb_used_data":768712,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281444960,"statfs":{"total":289910292480,"available":288199639040,"internally_reserved":0,"allocated":787161088,"data_stored":786391383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,6,10,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17657856,"num_objects":-4311,"num_object_clones":0,"num_object_copies":-8622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4311,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.276373"},"pg_stats":[{"pgid":"2.f","version":"41'14898","reported_seq":14971,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.530544+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:38.530544+0000","last_peered":"2024-09-18T12:01:38.530544+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:38.530544+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18997248,"num_objects":4638,"num_object_clones":0,"num_object_copies":9276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14898,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1039}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"41'14783","reported_seq":14943,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:38.488966+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:38.488966+0000","last_peered":"2024-09-18T12:01:38.488966+0000","last_clean":"2024-09-18T12:01:38.488966+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:38.488966+0000","last_undegraded":"2024-09-18T12:01:38.488966+0000","last_fullsized":"2024-09-18T12:01:38.488966+0000","mapping_epoch":29,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19042304,"num_objects":4649,"num_object_clones":0,"num_object_copies":9298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14846","reported_seq":15033,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.528666+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:38.528666+0000","last_peered":"2024-09-18T12:01:38.528666+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:38.528666+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19296280,"num_objects":4712,"num_object_clones":0,"num_object_copies":9424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4712,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11704,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1176}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14653","reported_seq":14684,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:38.500602+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:38.500602+0000","last_peered":"2024-09-18T12:01:38.500602+0000","last_clean":"2024-09-18T12:01:38.500602+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:38.500602+0000","last_undegraded":"2024-09-18T12:01:38.500602+0000","last_fullsized":"2024-09-18T12:01:38.500602+0000","mapping_epoch":29,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18870272,"num_objects":4607,"num_object_clones":0,"num_object_copies":9214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14653,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14739","reported_seq":14932,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:38.502828+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:38.502828+0000","last_peered":"2024-09-18T12:01:38.502828+0000","last_clean":"2024-09-18T12:01:38.502828+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:38.502828+0000","last_undegraded":"2024-09-18T12:01:38.502828+0000","last_fullsized":"2024-09-18T12:01:38.502828+0000","mapping_epoch":29,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19283968,"num_objects":4708,"num_object_clones":0,"num_object_copies":9416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11904,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14689","reported_seq":14772,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.560949+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:38.560949+0000","last_peered":"2024-09-18T12:01:38.560949+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:38.560949+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19391698,"num_objects":4736,"num_object_clones":0,"num_object_copies":9472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14687,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14718","reported_seq":14807,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.559598+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:38.559598+0000","last_peered":"2024-09-18T12:01:38.559598+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:38.559598+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19064018,"num_objects":4656,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14716,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1097}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'14973","reported_seq":15200,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:38.437733+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:38.437733+0000","last_peered":"2024-09-18T12:01:38.437733+0000","last_clean":"2024-09-18T12:01:38.437733+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:38.437733+0000","last_undegraded":"2024-09-18T12:01:38.437733+0000","last_fullsized":"2024-09-18T12:01:38.437733+0000","mapping_epoch":29,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18960384,"num_objects":4629,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11290,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720111+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T11:59:15.720111+0000","last_peered":"2024-09-18T11:59:15.720111+0000","last_clean":"2024-09-18T11:59:15.720111+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T11:59:15.720111+0000","last_undegraded":"2024-09-18T11:59:15.720111+0000","last_fullsized":"2024-09-18T11:59:15.720111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"41'14722","reported_seq":21852,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:39.563597+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:39.563597+0000","last_peered":"2024-09-18T12:01:39.563597+0000","last_clean":"2024-09-18T12:01:39.563597+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:39.563597+0000","last_undegraded":"2024-09-18T12:01:39.563597+0000","last_fullsized":"2024-09-18T12:01:39.563597+0000","mapping_epoch":38,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19182802,"num_objects":4685,"num_object_clones":0,"num_object_copies":9370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14720,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721124+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:15.721124+0000","last_peered":"2024-09-18T11:59:15.721124+0000","last_clean":"2024-09-18T11:59:15.721124+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:15.721124+0000","last_undegraded":"2024-09-18T11:59:15.721124+0000","last_fullsized":"2024-09-18T11:59:15.721124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"41'14754","reported_seq":14832,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.561987+0000","last_change":"2024-09-18T11:55:04.728869+0000","last_active":"2024-09-18T12:01:38.561987+0000","last_peered":"2024-09-18T12:01:38.561987+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:38.561987+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19219666,"num_objects":4694,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14752,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1162}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":93,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.709908+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:15.709908+0000","last_peered":"2024-09-18T11:59:15.709908+0000","last_clean":"2024-09-18T11:59:15.709908+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:15.709908+0000","last_undegraded":"2024-09-18T11:59:15.709908+0000","last_fullsized":"2024-09-18T11:59:15.709908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14802","reported_seq":14860,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.500604+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:38.500604+0000","last_peered":"2024-09-18T12:01:38.500604+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:38.500604+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19092690,"num_objects":4663,"num_object_clones":0,"num_object_copies":9326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14800,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1113}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":73,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720299+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T11:59:15.720299+0000","last_peered":"2024-09-18T11:59:15.720299+0000","last_clean":"2024-09-18T11:59:15.720299+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T11:59:15.720299+0000","last_undegraded":"2024-09-18T11:59:15.720299+0000","last_fullsized":"2024-09-18T11:59:15.720299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14833","reported_seq":21907,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.536981+0000","last_change":"2024-09-18T11:59:15.619746+0000","last_active":"2024-09-18T12:01:38.536981+0000","last_peered":"2024-09-18T12:01:38.536981+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:38.536981+0000","last_undegraded":"2024-09-18T11:55:04.308388+0000","last_fullsized":"2024-09-18T11:55:04.245085+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:23:26.545393+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19107840,"num_objects":4665,"num_object_clones":0,"num_object_copies":9330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11313,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3431,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":4501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"41'14801","reported_seq":14885,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.424432+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:38.424432+0000","last_peered":"2024-09-18T12:01:38.424432+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:38.424432+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19362409,"num_objects":4728,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14800,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1193}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721494+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T11:59:15.721494+0000","last_peered":"2024-09-18T11:59:15.721494+0000","last_clean":"2024-09-18T11:59:15.721494+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T11:59:15.721494+0000","last_undegraded":"2024-09-18T11:59:15.721494+0000","last_fullsized":"2024-09-18T11:59:15.721494+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"41'14889","reported_seq":22037,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:39.594940+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:39.594940+0000","last_peered":"2024-09-18T12:01:39.594940+0000","last_clean":"2024-09-18T12:01:39.594940+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:39.594940+0000","last_undegraded":"2024-09-18T12:01:39.594940+0000","last_fullsized":"2024-09-18T12:01:39.594940+0000","mapping_epoch":40,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18978619,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14886,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721310+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:15.721310+0000","last_peered":"2024-09-18T11:59:15.721310+0000","last_clean":"2024-09-18T11:59:15.721310+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:15.721310+0000","last_undegraded":"2024-09-18T11:59:15.721310+0000","last_fullsized":"2024-09-18T11:59:15.721310+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"41'14630","reported_seq":14713,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.383845+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:38.383845+0000","last_peered":"2024-09-18T12:01:38.383845+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:38.383845+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4600","ondisk_log_start":"37'4600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18711762,"num_objects":4570,"num_object_clones":0,"num_object_copies":9140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14628,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":977}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.721041+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T11:59:15.721041+0000","last_peered":"2024-09-18T11:59:15.721041+0000","last_clean":"2024-09-18T11:59:15.721041+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T11:59:15.721041+0000","last_undegraded":"2024-09-18T11:59:15.721041+0000","last_fullsized":"2024-09-18T11:59:15.721041+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":68,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T11:59:15.720293+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T11:59:15.720293+0000","last_peered":"2024-09-18T11:59:15.720293+0000","last_clean":"2024-09-18T11:59:15.720293+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T11:59:15.720293+0000","last_undegraded":"2024-09-18T11:59:15.720293+0000","last_fullsized":"2024-09-18T11:59:15.720293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"41'14838","reported_seq":14925,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:38.556756+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:38.556756+0000","last_peered":"2024-09-18T12:01:38.556756+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:38.556756+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19147172,"num_objects":4678,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14835,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":305709132,"num_objects":74654,"num_object_clones":0,"num_object_copies":149308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74654,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220561,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10459,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":784752640,"data_stored":784681874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160768,"ondisk_log_size":160768,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051677,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562020,"kb_used_data":264156,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93809820,"statfs":{"total":96636764160,"available":96061255680,"internally_reserved":0,"allocated":270495744,"data_stored":270257628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.436}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.32600000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855440,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525680,"kb_used_data":223656,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93846160,"statfs":{"total":96636764160,"available":96098467840,"internally_reserved":0,"allocated":229023744,"data_stored":228729821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.214},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.34899999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.379},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":6.4900000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582860,"kb_used_data":280900,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93788980,"statfs":{"total":96636764160,"available":96039915520,"internally_reserved":0,"allocated":287641600,"data_stored":287403934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":2.3290000000000002},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":2.161}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":2.516},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":287125504,"data_stored":287122025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228372480,"data_stored":228312744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":269254656,"data_stored":269247105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:47.122 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:47.904 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:47.915 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:47.927 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-18T12:01:47.206002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":297058399,"num_objects":72544,"num_object_clones":0,"num_object_copies":145088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72544,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222675,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10634,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160883,"ondisk_log_size":160883,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1669612,"kb_used_data":751116,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281445908,"statfs":{"total":289910292480,"available":288200609792,"internally_reserved":0,"allocated":769142784,"data_stored":768364290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,9,5,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9838592,"num_objects":-2402,"num_object_clones":0,"num_object_copies":-4804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2402,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001856"},"pg_stats":[{"pgid":"2.f","version":"41'15035","reported_seq":15109,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.478060+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:43.478060+0000","last_peered":"2024-09-18T12:01:43.478060+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:43.478060+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18436096,"num_objects":4501,"num_object_clones":0,"num_object_copies":9002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15035,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":902}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'14915","reported_seq":15076,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.496657+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:43.496657+0000","last_peered":"2024-09-18T12:01:43.496657+0000","last_clean":"2024-09-18T12:01:43.496657+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:43.496657+0000","last_undegraded":"2024-09-18T12:01:43.496657+0000","last_fullsized":"2024-09-18T12:01:43.496657+0000","mapping_epoch":29,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18501632,"num_objects":4517,"num_object_clones":0,"num_object_copies":9034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14977","reported_seq":15166,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.492205+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:43.492205+0000","last_peered":"2024-09-18T12:01:43.492205+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:43.492205+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18759704,"num_objects":4581,"num_object_clones":0,"num_object_copies":9162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11835,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14793","reported_seq":14825,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.469025+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:43.469025+0000","last_peered":"2024-09-18T12:01:43.469025+0000","last_clean":"2024-09-18T12:01:43.469025+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:43.469025+0000","last_undegraded":"2024-09-18T12:01:43.469025+0000","last_fullsized":"2024-09-18T12:01:43.469025+0000","mapping_epoch":29,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18296832,"num_objects":4467,"num_object_clones":0,"num_object_copies":8934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14793,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14881","reported_seq":15075,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.460956+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:43.460956+0000","last_peered":"2024-09-18T12:01:43.460956+0000","last_clean":"2024-09-18T12:01:43.460956+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:43.460956+0000","last_undegraded":"2024-09-18T12:01:43.460956+0000","last_fullsized":"2024-09-18T12:01:43.460956+0000","mapping_epoch":29,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18702336,"num_objects":4566,"num_object_clones":0,"num_object_copies":9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12046,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14809","reported_seq":14893,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.505007+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:43.505007+0000","last_peered":"2024-09-18T12:01:43.505007+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:43.505007+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18900178,"num_objects":4616,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14807,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14846","reported_seq":14936,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.486769+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:43.486769+0000","last_peered":"2024-09-18T12:01:43.486769+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:43.486769+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18539730,"num_objects":4528,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14844,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":969}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'15099","reported_seq":15327,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.371029+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:43.371029+0000","last_peered":"2024-09-18T12:01:43.371029+0000","last_clean":"2024-09-18T12:01:43.371029+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:43.371029+0000","last_undegraded":"2024-09-18T12:01:43.371029+0000","last_fullsized":"2024-09-18T12:01:43.371029+0000","mapping_epoch":29,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18444288,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11416,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'14838","reported_seq":21970,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.490132+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:44.490132+0000","last_peered":"2024-09-18T12:01:44.490132+0000","last_clean":"2024-09-18T12:01:44.490132+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:44.490132+0000","last_undegraded":"2024-09-18T12:01:44.490132+0000","last_fullsized":"2024-09-18T12:01:44.490132+0000","mapping_epoch":38,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18707666,"num_objects":4569,"num_object_clones":0,"num_object_copies":9138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14836,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":42,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:16.715104+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:16.715104+0000","last_peered":"2024-09-18T11:59:16.715104+0000","last_clean":"2024-09-18T11:59:16.715104+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:16.715104+0000","last_undegraded":"2024-09-18T11:59:16.715104+0000","last_fullsized":"2024-09-18T11:59:16.715104+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'14890","reported_seq":14991,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.557914+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:01:43.557914+0000","last_peered":"2024-09-18T12:01:43.557914+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:43.557914+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18662610,"num_objects":4558,"num_object_clones":0,"num_object_copies":9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14888,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1034}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":94,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:16.719899+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:16.719899+0000","last_peered":"2024-09-18T11:59:16.719899+0000","last_clean":"2024-09-18T11:59:16.719899+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:16.719899+0000","last_undegraded":"2024-09-18T11:59:16.719899+0000","last_fullsized":"2024-09-18T11:59:16.719899+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14953","reported_seq":15012,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.501470+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:43.501470+0000","last_peered":"2024-09-18T12:01:43.501470+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:43.501470+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18474194,"num_objects":4512,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14951,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":962}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14947","reported_seq":22352,"reported_epoch":42,"state":"peering","last_fresh":"2024-09-18T12:01:43.514904+0000","last_change":"2024-09-18T12:01:43.514904+0000","last_active":"2024-09-18T12:01:43.509816+0000","last_peered":"2024-09-18T12:01:43.461890+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:43.514904+0000","last_undegraded":"2024-09-18T12:01:43.514904+0000","last_fullsized":"2024-09-18T12:01:43.514904+0000","mapping_epoch":42,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18640896,"num_objects":4551,"num_object_clones":0,"num_object_copies":9102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11427,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3595,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'14938","reported_seq":15022,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.581092+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:43.581092+0000","last_peered":"2024-09-18T12:01:43.581092+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:43.581092+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18801257,"num_objects":4591,"num_object_clones":0,"num_object_copies":9182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14937,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1056}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15020","reported_seq":22170,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.589580+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:44.589580+0000","last_peered":"2024-09-18T12:01:44.589580+0000","last_clean":"2024-09-18T12:01:44.589580+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:44.589580+0000","last_undegraded":"2024-09-18T12:01:44.589580+0000","last_fullsized":"2024-09-18T12:01:44.589580+0000","mapping_epoch":40,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18442043,"num_objects":4505,"num_object_clones":0,"num_object_copies":9010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15017,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":42,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:16.718930+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:16.718930+0000","last_peered":"2024-09-18T11:59:16.718930+0000","last_clean":"2024-09-18T11:59:16.718930+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:16.718930+0000","last_undegraded":"2024-09-18T11:59:16.718930+0000","last_fullsized":"2024-09-18T11:59:16.718930+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'14764","reported_seq":14848,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.575326+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:43.575326+0000","last_peered":"2024-09-18T12:01:43.575326+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:43.575326+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18162898,"num_objects":4436,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14762,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'14975","reported_seq":15063,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.529448+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:43.529448+0000","last_peered":"2024-09-18T12:01:43.529448+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:43.529448+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18586020,"num_objects":4541,"num_object_clones":0,"num_object_copies":9082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14972,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":297058380,"num_objects":72542,"num_object_clones":0,"num_object_copies":145084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222673,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10633,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":767901696,"data_stored":767837888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160880,"ondisk_log_size":160880,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051678,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561032,"kb_used_data":259004,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93810808,"statfs":{"total":96636764160,"available":96062267392,"internally_reserved":0,"allocated":265220096,"data_stored":264976731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.17000000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":5.798}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":5.8479999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":12.712}]}]},{"osd":1,"up_from":35,"seq":150323855441,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528136,"kb_used_data":217792,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93843704,"statfs":{"total":96636764160,"available":96095952896,"internally_reserved":0,"allocated":223019008,"data_stored":222726794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.246}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.625},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.29799999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580444,"kb_used_data":274320,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93791396,"statfs":{"total":96636764160,"available":96042389504,"internally_reserved":0,"allocated":280903680,"data_stored":280660765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.32500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.308}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280469504,"data_stored":280466025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":222593024,"data_stored":222540246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":264839168,"data_stored":264831617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:47.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:48.659 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:48.665 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:48.680 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-18T12:01:47.206002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":297058399,"num_objects":72544,"num_object_clones":0,"num_object_copies":145088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72544,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222675,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10634,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160883,"ondisk_log_size":160883,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1669612,"kb_used_data":751116,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281445908,"statfs":{"total":289910292480,"available":288200609792,"internally_reserved":0,"allocated":769142784,"data_stored":768364290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,9,5,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9838592,"num_objects":-2402,"num_object_clones":0,"num_object_copies":-4804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2402,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001856"},"pg_stats":[{"pgid":"2.f","version":"41'15035","reported_seq":15109,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.478060+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:43.478060+0000","last_peered":"2024-09-18T12:01:43.478060+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:43.478060+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18436096,"num_objects":4501,"num_object_clones":0,"num_object_copies":9002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15035,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":902}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'14915","reported_seq":15076,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.496657+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:43.496657+0000","last_peered":"2024-09-18T12:01:43.496657+0000","last_clean":"2024-09-18T12:01:43.496657+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:43.496657+0000","last_undegraded":"2024-09-18T12:01:43.496657+0000","last_fullsized":"2024-09-18T12:01:43.496657+0000","mapping_epoch":29,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18501632,"num_objects":4517,"num_object_clones":0,"num_object_copies":9034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"41'14977","reported_seq":15166,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.492205+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:43.492205+0000","last_peered":"2024-09-18T12:01:43.492205+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:43.492205+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18759704,"num_objects":4581,"num_object_clones":0,"num_object_copies":9162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11835,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1045}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"41'14793","reported_seq":14825,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.469025+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:43.469025+0000","last_peered":"2024-09-18T12:01:43.469025+0000","last_clean":"2024-09-18T12:01:43.469025+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:43.469025+0000","last_undegraded":"2024-09-18T12:01:43.469025+0000","last_fullsized":"2024-09-18T12:01:43.469025+0000","mapping_epoch":29,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18296832,"num_objects":4467,"num_object_clones":0,"num_object_copies":8934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14793,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"41'14881","reported_seq":15075,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.460956+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:43.460956+0000","last_peered":"2024-09-18T12:01:43.460956+0000","last_clean":"2024-09-18T12:01:43.460956+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:43.460956+0000","last_undegraded":"2024-09-18T12:01:43.460956+0000","last_fullsized":"2024-09-18T12:01:43.460956+0000","mapping_epoch":29,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18702336,"num_objects":4566,"num_object_clones":0,"num_object_copies":9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12046,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"41'14809","reported_seq":14893,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.505007+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:43.505007+0000","last_peered":"2024-09-18T12:01:43.505007+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:43.505007+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18900178,"num_objects":4616,"num_object_clones":0,"num_object_copies":9232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14807,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1089}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"41'14846","reported_seq":14936,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.486769+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:43.486769+0000","last_peered":"2024-09-18T12:01:43.486769+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:43.486769+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18539730,"num_objects":4528,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14844,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":969}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"41'15099","reported_seq":15327,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.371029+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:43.371029+0000","last_peered":"2024-09-18T12:01:43.371029+0000","last_clean":"2024-09-18T12:01:43.371029+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:43.371029+0000","last_undegraded":"2024-09-18T12:01:43.371029+0000","last_fullsized":"2024-09-18T12:01:43.371029+0000","mapping_epoch":29,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18444288,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11416,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'14838","reported_seq":21970,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.490132+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:44.490132+0000","last_peered":"2024-09-18T12:01:44.490132+0000","last_clean":"2024-09-18T12:01:44.490132+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:44.490132+0000","last_undegraded":"2024-09-18T12:01:44.490132+0000","last_fullsized":"2024-09-18T12:01:44.490132+0000","mapping_epoch":38,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18707666,"num_objects":4569,"num_object_clones":0,"num_object_copies":9138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14836,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":42,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:16.715104+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T11:59:16.715104+0000","last_peered":"2024-09-18T11:59:16.715104+0000","last_clean":"2024-09-18T11:59:16.715104+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T11:59:16.715104+0000","last_undegraded":"2024-09-18T11:59:16.715104+0000","last_fullsized":"2024-09-18T11:59:16.715104+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'14890","reported_seq":14991,"reported_epoch":42,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.557914+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:01:43.557914+0000","last_peered":"2024-09-18T12:01:43.557914+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:43.557914+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18662610,"num_objects":4558,"num_object_clones":0,"num_object_copies":9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14888,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1034}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":94,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:16.719899+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T11:59:16.719899+0000","last_peered":"2024-09-18T11:59:16.719899+0000","last_clean":"2024-09-18T11:59:16.719899+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T11:59:16.719899+0000","last_undegraded":"2024-09-18T11:59:16.719899+0000","last_fullsized":"2024-09-18T11:59:16.719899+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"41'14953","reported_seq":15012,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.501470+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:43.501470+0000","last_peered":"2024-09-18T12:01:43.501470+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:43.501470+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18474194,"num_objects":4512,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14951,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":962}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"41'14947","reported_seq":22352,"reported_epoch":42,"state":"peering","last_fresh":"2024-09-18T12:01:43.514904+0000","last_change":"2024-09-18T12:01:43.514904+0000","last_active":"2024-09-18T12:01:43.509816+0000","last_peered":"2024-09-18T12:01:43.461890+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T11:55:04.665998+0000","last_became_peered":"2024-09-18T11:55:04.665998+0000","last_unstale":"2024-09-18T12:01:43.514904+0000","last_undegraded":"2024-09-18T12:01:43.514904+0000","last_fullsized":"2024-09-18T12:01:43.514904+0000","mapping_epoch":42,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18640896,"num_objects":4551,"num_object_clones":0,"num_object_copies":9102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11427,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3595,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'14938","reported_seq":15022,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.581092+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:43.581092+0000","last_peered":"2024-09-18T12:01:43.581092+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:43.581092+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18801257,"num_objects":4591,"num_object_clones":0,"num_object_copies":9182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14937,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1056}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15020","reported_seq":22170,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.589580+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:44.589580+0000","last_peered":"2024-09-18T12:01:44.589580+0000","last_clean":"2024-09-18T12:01:44.589580+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:44.589580+0000","last_undegraded":"2024-09-18T12:01:44.589580+0000","last_fullsized":"2024-09-18T12:01:44.589580+0000","mapping_epoch":40,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18442043,"num_objects":4505,"num_object_clones":0,"num_object_copies":9010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15017,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":42,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T11:59:16.718930+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T11:59:16.718930+0000","last_peered":"2024-09-18T11:59:16.718930+0000","last_clean":"2024-09-18T11:59:16.718930+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T11:59:16.718930+0000","last_undegraded":"2024-09-18T11:59:16.718930+0000","last_fullsized":"2024-09-18T11:59:16.718930+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'14764","reported_seq":14848,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.575326+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:43.575326+0000","last_peered":"2024-09-18T12:01:43.575326+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:43.575326+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4700","ondisk_log_start":"37'4700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18162898,"num_objects":4436,"num_object_clones":0,"num_object_copies":8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14762,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'14975","reported_seq":15063,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:43.529448+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:43.529448+0000","last_peered":"2024-09-18T12:01:43.529448+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:43.529448+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18586020,"num_objects":4541,"num_object_clones":0,"num_object_copies":9082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14972,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":297058380,"num_objects":72542,"num_object_clones":0,"num_object_copies":145084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222673,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10633,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":767901696,"data_stored":767837888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160880,"ondisk_log_size":160880,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051678,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561032,"kb_used_data":259004,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93810808,"statfs":{"total":96636764160,"available":96062267392,"internally_reserved":0,"allocated":265220096,"data_stored":264976731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.17000000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":5.798}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":5.8479999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":12.712}]}]},{"osd":1,"up_from":35,"seq":150323855441,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528136,"kb_used_data":217792,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93843704,"statfs":{"total":96636764160,"available":96095952896,"internally_reserved":0,"allocated":223019008,"data_stored":222726794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.246}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.625},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.29799999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580444,"kb_used_data":274320,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93791396,"statfs":{"total":96636764160,"available":96042389504,"internally_reserved":0,"allocated":280903680,"data_stored":280660765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.32500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.308}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280469504,"data_stored":280466025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":222593024,"data_stored":222540246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":264839168,"data_stored":264831617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:48.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:01:48.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "41'15035", 'reported_seq': 15109, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:43.478060+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:01:43.478060+0000', 'last_peered': '2024-09-18T12:01:43.478060+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:01:43.478060+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'5000", 'ondisk_log_start': "37'5000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18436096, 'num_objects': 4501, 'num_object_clones': 0, 'num_object_copies': 9002, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15035, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 902}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:48.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:01:48.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "41'14977", 'reported_seq': 15166, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:43.492205+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:01:43.492205+0000', 'last_peered': '2024-09-18T12:01:43.492205+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:01:43.492205+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'4900", 'ondisk_log_start': "37'4900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18759704, 'num_objects': 4581, 'num_object_clones': 0, 'num_object_copies': 9162, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4581, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11835, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:48.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:01:48.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "41'14809", 'reported_seq': 14893, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:43.505007+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:01:43.505007+0000', 'last_peered': '2024-09-18T12:01:43.505007+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:01:43.505007+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'4800", 'ondisk_log_start': "37'4800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18900178, 'num_objects': 4616, 'num_object_clones': 0, 'num_object_copies': 9232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14807, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1089}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:48.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:01:48.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "41'14846", 'reported_seq': 14936, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:43.486769+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:01:43.486769+0000', 'last_peered': '2024-09-18T12:01:43.486769+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:01:43.486769+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'4800", 'ondisk_log_start': "37'4800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18539730, 'num_objects': 4528, 'num_object_clones': 0, 'num_object_copies': 9056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4528, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14844, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 969}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:48.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:01:48.682 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'14890", 'reported_seq': 14991, 'reported_epoch': 42, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:43.557914+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:01:43.557914+0000', 'last_peered': '2024-09-18T12:01:43.557914+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:01:43.557914+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'4800", 'ondisk_log_start': "37'4800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18662610, 'num_objects': 4558, 'num_object_clones': 0, 'num_object_copies': 9116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3524, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14888, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1034}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:48.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:01:48.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "41'14953", 'reported_seq': 15012, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:43.501470+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:01:43.501470+0000', 'last_peered': '2024-09-18T12:01:43.501470+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:01:43.501470+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'4900", 'ondisk_log_start': "37'4900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18474194, 'num_objects': 4512, 'num_object_clones': 0, 'num_object_copies': 9024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14951, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 962}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:48.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:01:48.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "41'14947", 'reported_seq': 22352, 'reported_epoch': 42, 'state': 'peering', 'last_fresh': '2024-09-18T12:01:43.514904+0000', 'last_change': '2024-09-18T12:01:43.514904+0000', 'last_active': '2024-09-18T12:01:43.509816+0000', 'last_peered': '2024-09-18T12:01:43.461890+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T11:55:04.665998+0000', 'last_became_peered': '2024-09-18T11:55:04.665998+0000', 'last_unstale': '2024-09-18T12:01:43.514904+0000', 'last_undegraded': '2024-09-18T12:01:43.514904+0000', 'last_fullsized': '2024-09-18T12:01:43.514904+0000', 'mapping_epoch': 42, 'log_start': "37'4900", 'ondisk_log_start': "37'4900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18640896, 'num_objects': 4551, 'num_object_clones': 0, 'num_object_copies': 9102, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4551, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11427, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3595, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:48.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:01:48.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'14938", 'reported_seq': 15022, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:43.581092+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:01:43.581092+0000', 'last_peered': '2024-09-18T12:01:43.581092+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:01:43.581092+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'4900", 'ondisk_log_start': "37'4900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18801257, 'num_objects': 4591, 'num_object_clones': 0, 'num_object_copies': 9182, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14937, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1056}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:48.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:01:48.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'14764", 'reported_seq': 14848, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:43.575326+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:01:43.575326+0000', 'last_peered': '2024-09-18T12:01:43.575326+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:01:43.575326+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'4700", 'ondisk_log_start': "37'4700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18162898, 'num_objects': 4436, 'num_object_clones': 0, 'num_object_copies': 8872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4436, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14762, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 843}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:48.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:01:48.683 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'14975", 'reported_seq': 15063, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:43.529448+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:01:43.529448+0000', 'last_peered': '2024-09-18T12:01:43.529448+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:01:43.529448+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'4900", 'ondisk_log_start': "37'4900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18586020, 'num_objects': 4541, 'num_object_clones': 0, 'num_object_copies': 9082, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14972, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1018}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:48.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:01:49.704 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:49.704 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31847/141714 objects degraded (22.473%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":509,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":70857,"data_bytes":290148447,"bytes_used":1708531712,"bytes_avail":288201760768,"bytes_total":289910292480,"degraded_objects":31847,"degraded_total":141714,"degraded_ratio":0.22472726759529757,"recovering_objects_per_sec":35,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":287,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":474},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:01:49.757 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:01:49.758 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:50.533 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:50.547 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:50.582 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-18T12:01:49.206401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":290148447,"num_objects":70857,"num_object_clones":0,"num_object_copies":141714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70857,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224362,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10747,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160670,"ondisk_log_size":160670,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1668488,"kb_used_data":741656,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281447032,"statfs":{"total":289910292480,"available":288201760768,"internally_reserved":0,"allocated":759455744,"data_stored":758666710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,7,6,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15560704,"num_objects":-3799,"num_object_clones":0,"num_object_copies":-7598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3799,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001776"},"pg_stats":[{"pgid":"2.f","version":"43'15155","reported_seq":15227,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.531669+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:48.531669+0000","last_peered":"2024-09-18T12:01:48.531669+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:48.531669+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17944576,"num_objects":4381,"num_object_clones":0,"num_object_copies":8762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15155,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":782}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15023","reported_seq":15185,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:48.513311+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:48.513311+0000","last_peered":"2024-09-18T12:01:48.513311+0000","last_clean":"2024-09-18T12:01:48.513311+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:48.513311+0000","last_undegraded":"2024-09-18T12:01:48.513311+0000","last_fullsized":"2024-09-18T12:01:48.513311+0000","mapping_epoch":29,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18059264,"num_objects":4409,"num_object_clones":0,"num_object_copies":8818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12215,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15104","reported_seq":15294,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.479744+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:48.479744+0000","last_peered":"2024-09-18T12:01:48.479744+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:48.479744+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18239512,"num_objects":4454,"num_object_clones":0,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11962,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":918}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'14911","reported_seq":14944,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:48.436310+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:48.436310+0000","last_peered":"2024-09-18T12:01:48.436310+0000","last_clean":"2024-09-18T12:01:48.436310+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:48.436310+0000","last_undegraded":"2024-09-18T12:01:48.436310+0000","last_fullsized":"2024-09-18T12:01:48.436310+0000","mapping_epoch":29,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17813504,"num_objects":4349,"num_object_clones":0,"num_object_copies":8698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14911,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'14994","reported_seq":15189,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:48.577139+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:48.577139+0000","last_peered":"2024-09-18T12:01:48.577139+0000","last_clean":"2024-09-18T12:01:48.577139+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:48.577139+0000","last_undegraded":"2024-09-18T12:01:48.577139+0000","last_fullsized":"2024-09-18T12:01:48.577139+0000","mapping_epoch":29,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18239488,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12159,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'14927","reported_seq":15012,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.484131+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:48.484131+0000","last_peered":"2024-09-18T12:01:48.484131+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:48.484131+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18416850,"num_objects":4498,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14925,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":971}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'14967","reported_seq":15058,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.537520+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:48.537520+0000","last_peered":"2024-09-18T12:01:48.537520+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:48.537520+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18044114,"num_objects":4407,"num_object_clones":0,"num_object_copies":8814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14965,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15220","reported_seq":15449,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:48.562038+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:48.562038+0000","last_peered":"2024-09-18T12:01:48.562038+0000","last_clean":"2024-09-18T12:01:48.562038+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:48.562038+0000","last_undegraded":"2024-09-18T12:01:48.562038+0000","last_fullsized":"2024-09-18T12:01:48.562038+0000","mapping_epoch":29,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17948672,"num_objects":4382,"num_object_clones":0,"num_object_copies":8764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11537,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'14838","reported_seq":21970,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.490132+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:44.490132+0000","last_peered":"2024-09-18T12:01:44.490132+0000","last_clean":"2024-09-18T12:01:44.490132+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:44.490132+0000","last_undegraded":"2024-09-18T12:01:44.490132+0000","last_fullsized":"2024-09-18T12:01:44.490132+0000","mapping_epoch":38,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18707666,"num_objects":4569,"num_object_clones":0,"num_object_copies":9138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14836,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15032","reported_seq":15359,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.571624+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:01:48.571624+0000","last_peered":"2024-09-18T12:01:48.571624+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:48.571624+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18080978,"num_objects":4416,"num_object_clones":0,"num_object_copies":8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15030,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1004}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15072","reported_seq":15132,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.476824+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:48.476824+0000","last_peered":"2024-09-18T12:01:48.476824+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:48.476824+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17986770,"num_objects":4393,"num_object_clones":0,"num_object_copies":8786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15070,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":843}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15076","reported_seq":22489,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:01:48.528125+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:01:48.528125+0000","last_peered":"2024-09-18T12:01:48.528125+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:01:48.528125+0000","last_undegraded":"2024-09-18T12:01:48.528125+0000","last_fullsized":"2024-09-18T12:01:48.528125+0000","mapping_epoch":42,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18112512,"num_objects":4422,"num_object_clones":0,"num_object_copies":8844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":4422}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15041","reported_seq":15126,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.564293+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:48.564293+0000","last_peered":"2024-09-18T12:01:48.564293+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:48.564293+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18379369,"num_objects":4488,"num_object_clones":0,"num_object_copies":8976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15040,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":953}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15020","reported_seq":22170,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.589580+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:44.589580+0000","last_peered":"2024-09-18T12:01:44.589580+0000","last_clean":"2024-09-18T12:01:44.589580+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:44.589580+0000","last_undegraded":"2024-09-18T12:01:44.589580+0000","last_fullsized":"2024-09-18T12:01:44.589580+0000","mapping_epoch":40,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18442043,"num_objects":4505,"num_object_clones":0,"num_object_copies":9010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15017,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'14881","reported_seq":14966,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.536361+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:48.536361+0000","last_peered":"2024-09-18T12:01:48.536361+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:48.536361+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17683666,"num_objects":4319,"num_object_clones":0,"num_object_copies":8638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14879,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15106","reported_seq":15195,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.563760+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:48.563760+0000","last_peered":"2024-09-18T12:01:48.563760+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:48.563760+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18049444,"num_objects":4410,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15103,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":290148428,"num_objects":70855,"num_object_clones":0,"num_object_copies":141710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70855,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224360,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10746,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":758108160,"data_stored":758044352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160667,"ondisk_log_size":160667,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051679,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561368,"kb_used_data":255172,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93810472,"statfs":{"total":96636764160,"available":96061923328,"internally_reserved":0,"allocated":261296128,"data_stored":261047493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.42599999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":7.79}]}]},{"osd":1,"up_from":35,"seq":150323855441,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528136,"kb_used_data":217792,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93843704,"statfs":{"total":96636764160,"available":96095952896,"internally_reserved":0,"allocated":223019008,"data_stored":222726794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.246}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.625},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.29799999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578984,"kb_used_data":268692,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93792856,"statfs":{"total":96636764160,"available":96043884544,"internally_reserved":0,"allocated":275140608,"data_stored":274892423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.1900000000000004},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23400000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":4.4180000000000001},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":3.9430000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274620416,"data_stored":274616937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":222593024,"data_stored":222540246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":260894720,"data_stored":260887169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:53.584 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:54.351 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:54.379 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:54.393 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-18T12:01:53.211388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":289042527,"num_objects":70587,"num_object_clones":0,"num_object_copies":141174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70587,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224632,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10747,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1649608,"kb_used_data":735256,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281465912,"statfs":{"total":289910292480,"available":288221093888,"internally_reserved":0,"allocated":752902144,"data_stored":752107840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,4,9,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16666624,"num_objects":-4069,"num_object_clones":0,"num_object_copies":-8138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4069,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.732152"},"pg_stats":[{"pgid":"2.f","version":"43'15155","reported_seq":15227,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.531669+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:48.531669+0000","last_peered":"2024-09-18T12:01:48.531669+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:48.531669+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17944576,"num_objects":4381,"num_object_clones":0,"num_object_copies":8762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15155,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":782}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15023","reported_seq":15185,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:48.513311+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:48.513311+0000","last_peered":"2024-09-18T12:01:48.513311+0000","last_clean":"2024-09-18T12:01:48.513311+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:48.513311+0000","last_undegraded":"2024-09-18T12:01:48.513311+0000","last_fullsized":"2024-09-18T12:01:48.513311+0000","mapping_epoch":29,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18059264,"num_objects":4409,"num_object_clones":0,"num_object_copies":8818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12215,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15104","reported_seq":15294,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.479744+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:48.479744+0000","last_peered":"2024-09-18T12:01:48.479744+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:48.479744+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18239512,"num_objects":4454,"num_object_clones":0,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11962,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":918}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'14911","reported_seq":14944,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:48.436310+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:48.436310+0000","last_peered":"2024-09-18T12:01:48.436310+0000","last_clean":"2024-09-18T12:01:48.436310+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:48.436310+0000","last_undegraded":"2024-09-18T12:01:48.436310+0000","last_fullsized":"2024-09-18T12:01:48.436310+0000","mapping_epoch":29,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17813504,"num_objects":4349,"num_object_clones":0,"num_object_copies":8698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14911,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'14994","reported_seq":15189,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:48.577139+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:48.577139+0000","last_peered":"2024-09-18T12:01:48.577139+0000","last_clean":"2024-09-18T12:01:48.577139+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:48.577139+0000","last_undegraded":"2024-09-18T12:01:48.577139+0000","last_fullsized":"2024-09-18T12:01:48.577139+0000","mapping_epoch":29,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18239488,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12159,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'14927","reported_seq":15012,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.484131+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:48.484131+0000","last_peered":"2024-09-18T12:01:48.484131+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:48.484131+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18416850,"num_objects":4498,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14925,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":971}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'14967","reported_seq":15058,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.537520+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:48.537520+0000","last_peered":"2024-09-18T12:01:48.537520+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:48.537520+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18044114,"num_objects":4407,"num_object_clones":0,"num_object_copies":8814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14965,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15220","reported_seq":15449,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:48.562038+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:48.562038+0000","last_peered":"2024-09-18T12:01:48.562038+0000","last_clean":"2024-09-18T12:01:48.562038+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:48.562038+0000","last_undegraded":"2024-09-18T12:01:48.562038+0000","last_fullsized":"2024-09-18T12:01:48.562038+0000","mapping_epoch":29,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17948672,"num_objects":4382,"num_object_clones":0,"num_object_copies":8764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11537,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'14978","reported_seq":22110,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:49.679946+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:49.679946+0000","last_peered":"2024-09-18T12:01:49.679946+0000","last_clean":"2024-09-18T12:01:49.679946+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:49.679946+0000","last_undegraded":"2024-09-18T12:01:49.679946+0000","last_fullsized":"2024-09-18T12:01:49.679946+0000","mapping_epoch":38,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18134226,"num_objects":4429,"num_object_clones":0,"num_object_copies":8858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14976,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15032","reported_seq":15359,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.571624+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:01:48.571624+0000","last_peered":"2024-09-18T12:01:48.571624+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:48.571624+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18080978,"num_objects":4416,"num_object_clones":0,"num_object_copies":8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15030,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1004}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15072","reported_seq":15132,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.476824+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:48.476824+0000","last_peered":"2024-09-18T12:01:48.476824+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:48.476824+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17986770,"num_objects":4393,"num_object_clones":0,"num_object_copies":8786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15070,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":843}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15076","reported_seq":22489,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:01:48.528125+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:01:48.528125+0000","last_peered":"2024-09-18T12:01:48.528125+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:01:48.528125+0000","last_undegraded":"2024-09-18T12:01:48.528125+0000","last_fullsized":"2024-09-18T12:01:48.528125+0000","mapping_epoch":42,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18112512,"num_objects":4422,"num_object_clones":0,"num_object_copies":8844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":4422}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15041","reported_seq":15126,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.564293+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:48.564293+0000","last_peered":"2024-09-18T12:01:48.564293+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:48.564293+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18379369,"num_objects":4488,"num_object_clones":0,"num_object_copies":8976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15040,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":953}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15150","reported_seq":22300,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:49.699813+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:49.699813+0000","last_peered":"2024-09-18T12:01:49.699813+0000","last_clean":"2024-09-18T12:01:49.699813+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:49.699813+0000","last_undegraded":"2024-09-18T12:01:49.699813+0000","last_fullsized":"2024-09-18T12:01:49.699813+0000","mapping_epoch":40,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17909563,"num_objects":4375,"num_object_clones":0,"num_object_copies":8750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15147,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'14881","reported_seq":14966,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.536361+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:48.536361+0000","last_peered":"2024-09-18T12:01:48.536361+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:48.536361+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17683666,"num_objects":4319,"num_object_clones":0,"num_object_copies":8638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14879,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15106","reported_seq":15195,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.563760+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:48.563760+0000","last_peered":"2024-09-18T12:01:48.563760+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:48.563760+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18049444,"num_objects":4410,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15103,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":289042508,"num_objects":70585,"num_object_clones":0,"num_object_copies":141170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70585,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224630,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10746,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":751210496,"data_stored":751146688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160737,"ondisk_log_size":160737,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051679,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561368,"kb_used_data":255172,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93810472,"statfs":{"total":96636764160,"available":96061923328,"internally_reserved":0,"allocated":261296128,"data_stored":261047493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.42599999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":7.79}]}]},{"osd":1,"up_from":35,"seq":150323855442,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509256,"kb_used_data":211392,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93862584,"statfs":{"total":96636764160,"available":96115286016,"internally_reserved":0,"allocated":216465408,"data_stored":216167924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,0,5,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":4.4850000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":4.2549999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.30299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578984,"kb_used_data":268692,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93792856,"statfs":{"total":96636764160,"available":96043884544,"internally_reserved":0,"allocated":275140608,"data_stored":274892423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.1900000000000004},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23400000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":4.4180000000000001},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":3.9430000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274620416,"data_stored":274616937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":215695360,"data_stored":215642582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":260894720,"data_stored":260887169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:54.394 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:55.022 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:55.056 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:55.084 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-18T12:01:53.211388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":289042527,"num_objects":70587,"num_object_clones":0,"num_object_copies":141174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70587,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224632,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10747,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1649608,"kb_used_data":735256,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281465912,"statfs":{"total":289910292480,"available":288221093888,"internally_reserved":0,"allocated":752902144,"data_stored":752107840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,4,9,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16666624,"num_objects":-4069,"num_object_clones":0,"num_object_copies":-8138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4069,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.732152"},"pg_stats":[{"pgid":"2.f","version":"43'15155","reported_seq":15227,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.531669+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:48.531669+0000","last_peered":"2024-09-18T12:01:48.531669+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:48.531669+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17944576,"num_objects":4381,"num_object_clones":0,"num_object_copies":8762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15155,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":782}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15023","reported_seq":15185,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:48.513311+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:48.513311+0000","last_peered":"2024-09-18T12:01:48.513311+0000","last_clean":"2024-09-18T12:01:48.513311+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:48.513311+0000","last_undegraded":"2024-09-18T12:01:48.513311+0000","last_fullsized":"2024-09-18T12:01:48.513311+0000","mapping_epoch":29,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18059264,"num_objects":4409,"num_object_clones":0,"num_object_copies":8818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12215,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15104","reported_seq":15294,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.479744+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:48.479744+0000","last_peered":"2024-09-18T12:01:48.479744+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:48.479744+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18239512,"num_objects":4454,"num_object_clones":0,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11962,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":918}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'14911","reported_seq":14944,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:48.436310+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:48.436310+0000","last_peered":"2024-09-18T12:01:48.436310+0000","last_clean":"2024-09-18T12:01:48.436310+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:48.436310+0000","last_undegraded":"2024-09-18T12:01:48.436310+0000","last_fullsized":"2024-09-18T12:01:48.436310+0000","mapping_epoch":29,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17813504,"num_objects":4349,"num_object_clones":0,"num_object_copies":8698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14911,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'14994","reported_seq":15189,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:48.577139+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:48.577139+0000","last_peered":"2024-09-18T12:01:48.577139+0000","last_clean":"2024-09-18T12:01:48.577139+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:48.577139+0000","last_undegraded":"2024-09-18T12:01:48.577139+0000","last_fullsized":"2024-09-18T12:01:48.577139+0000","mapping_epoch":29,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18239488,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12159,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'14927","reported_seq":15012,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.484131+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:48.484131+0000","last_peered":"2024-09-18T12:01:48.484131+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:48.484131+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18416850,"num_objects":4498,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14925,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":971}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'14967","reported_seq":15058,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.537520+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:48.537520+0000","last_peered":"2024-09-18T12:01:48.537520+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:48.537520+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18044114,"num_objects":4407,"num_object_clones":0,"num_object_copies":8814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14965,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15220","reported_seq":15449,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:48.562038+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:48.562038+0000","last_peered":"2024-09-18T12:01:48.562038+0000","last_clean":"2024-09-18T12:01:48.562038+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:48.562038+0000","last_undegraded":"2024-09-18T12:01:48.562038+0000","last_fullsized":"2024-09-18T12:01:48.562038+0000","mapping_epoch":29,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17948672,"num_objects":4382,"num_object_clones":0,"num_object_copies":8764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11537,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'14978","reported_seq":22110,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:49.679946+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:49.679946+0000","last_peered":"2024-09-18T12:01:49.679946+0000","last_clean":"2024-09-18T12:01:49.679946+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:49.679946+0000","last_undegraded":"2024-09-18T12:01:49.679946+0000","last_fullsized":"2024-09-18T12:01:49.679946+0000","mapping_epoch":38,"log_start":"37'4900","ondisk_log_start":"37'4900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18134226,"num_objects":4429,"num_object_clones":0,"num_object_copies":8858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14976,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15032","reported_seq":15359,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.571624+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:01:48.571624+0000","last_peered":"2024-09-18T12:01:48.571624+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:48.571624+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18080978,"num_objects":4416,"num_object_clones":0,"num_object_copies":8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15030,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1004}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15072","reported_seq":15132,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.476824+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:48.476824+0000","last_peered":"2024-09-18T12:01:48.476824+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:48.476824+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17986770,"num_objects":4393,"num_object_clones":0,"num_object_copies":8786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15070,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":843}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15076","reported_seq":22489,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:01:48.528125+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:01:48.528125+0000","last_peered":"2024-09-18T12:01:48.528125+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:01:48.528125+0000","last_undegraded":"2024-09-18T12:01:48.528125+0000","last_fullsized":"2024-09-18T12:01:48.528125+0000","mapping_epoch":42,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18112512,"num_objects":4422,"num_object_clones":0,"num_object_copies":8844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":4422}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15041","reported_seq":15126,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.564293+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:48.564293+0000","last_peered":"2024-09-18T12:01:48.564293+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:48.564293+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18379369,"num_objects":4488,"num_object_clones":0,"num_object_copies":8976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15040,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":953}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15150","reported_seq":22300,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:49.699813+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:49.699813+0000","last_peered":"2024-09-18T12:01:49.699813+0000","last_clean":"2024-09-18T12:01:49.699813+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:49.699813+0000","last_undegraded":"2024-09-18T12:01:49.699813+0000","last_fullsized":"2024-09-18T12:01:49.699813+0000","mapping_epoch":40,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17909563,"num_objects":4375,"num_object_clones":0,"num_object_copies":8750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15147,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'14881","reported_seq":14966,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.536361+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:48.536361+0000","last_peered":"2024-09-18T12:01:48.536361+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:48.536361+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'4800","ondisk_log_start":"37'4800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17683666,"num_objects":4319,"num_object_clones":0,"num_object_copies":8638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14879,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15106","reported_seq":15195,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:48.563760+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:48.563760+0000","last_peered":"2024-09-18T12:01:48.563760+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:48.563760+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18049444,"num_objects":4410,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15103,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":289042508,"num_objects":70585,"num_object_clones":0,"num_object_copies":141170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70585,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224630,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10746,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":751210496,"data_stored":751146688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160737,"ondisk_log_size":160737,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051679,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561368,"kb_used_data":255172,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93810472,"statfs":{"total":96636764160,"available":96061923328,"internally_reserved":0,"allocated":261296128,"data_stored":261047493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.42599999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":7.79}]}]},{"osd":1,"up_from":35,"seq":150323855442,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509256,"kb_used_data":211392,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93862584,"statfs":{"total":96636764160,"available":96115286016,"internally_reserved":0,"allocated":216465408,"data_stored":216167924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,0,5,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":4.4850000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":4.2549999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.30299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578984,"kb_used_data":268692,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93792856,"statfs":{"total":96636764160,"available":96043884544,"internally_reserved":0,"allocated":275140608,"data_stored":274892423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":4.1900000000000004},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23400000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":4.4180000000000001},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":3.9430000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274620416,"data_stored":274616937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":215695360,"data_stored":215642582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":260894720,"data_stored":260887169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:55.085 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:01:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'15155", 'reported_seq': 15227, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:48.531669+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:01:48.531669+0000', 'last_peered': '2024-09-18T12:01:48.531669+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:01:48.531669+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'5100", 'ondisk_log_start': "37'5100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17944576, 'num_objects': 4381, 'num_object_clones': 0, 'num_object_copies': 8762, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15155, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 782}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:01:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'15104", 'reported_seq': 15294, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:48.479744+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:01:48.479744+0000', 'last_peered': '2024-09-18T12:01:48.479744+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:01:48.479744+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'5100", 'ondisk_log_start': "37'5100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18239512, 'num_objects': 4454, 'num_object_clones': 0, 'num_object_copies': 8908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4454, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11962, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 918}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:01:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'14927", 'reported_seq': 15012, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:48.484131+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:01:48.484131+0000', 'last_peered': '2024-09-18T12:01:48.484131+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:01:48.484131+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'4900", 'ondisk_log_start': "37'4900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18416850, 'num_objects': 4498, 'num_object_clones': 0, 'num_object_copies': 8996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14925, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 971}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:01:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'14967", 'reported_seq': 15058, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:48.537520+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:01:48.537520+0000', 'last_peered': '2024-09-18T12:01:48.537520+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:01:48.537520+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'4900", 'ondisk_log_start': "37'4900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18044114, 'num_objects': 4407, 'num_object_clones': 0, 'num_object_copies': 8814, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14965, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 848}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:01:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'15032", 'reported_seq': 15359, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:48.571624+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:01:48.571624+0000', 'last_peered': '2024-09-18T12:01:48.571624+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:01:48.571624+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'5000", 'ondisk_log_start': "37'5000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18080978, 'num_objects': 4416, 'num_object_clones': 0, 'num_object_copies': 8832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3412, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15030, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 124, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1004}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:55.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:01:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'15072", 'reported_seq': 15132, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:48.476824+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:01:48.476824+0000', 'last_peered': '2024-09-18T12:01:48.476824+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:01:48.476824+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'5000", 'ondisk_log_start': "37'5000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17986770, 'num_objects': 4393, 'num_object_clones': 0, 'num_object_copies': 8786, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15070, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 843}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:01:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'15076", 'reported_seq': 22489, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:01:48.528125+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:01:48.528125+0000', 'last_peered': '2024-09-18T12:01:48.528125+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:01:48.528125+0000', 'last_undegraded': '2024-09-18T12:01:48.528125+0000', 'last_fullsized': '2024-09-18T12:01:48.528125+0000', 'mapping_epoch': 42, 'log_start': "37'5000", 'ondisk_log_start': "37'5000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18112512, 'num_objects': 4422, 'num_object_clones': 0, 'num_object_copies': 8844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4422, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11556, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 4422}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:01:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:01:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'15041", 'reported_seq': 15126, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:48.564293+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:01:48.564293+0000', 'last_peered': '2024-09-18T12:01:48.564293+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:01:48.564293+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'5000", 'ondisk_log_start': "37'5000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18379369, 'num_objects': 4488, 'num_object_clones': 0, 'num_object_copies': 8976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15040, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 953}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:01:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'14881", 'reported_seq': 14966, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:48.536361+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:01:48.536361+0000', 'last_peered': '2024-09-18T12:01:48.536361+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:01:48.536361+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'4800", 'ondisk_log_start': "37'4800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17683666, 'num_objects': 4319, 'num_object_clones': 0, 'num_object_copies': 8638, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14879, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 726}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:01:55.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'15106", 'reported_seq': 15195, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:48.563760+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:01:48.563760+0000', 'last_peered': '2024-09-18T12:01:48.563760+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:01:48.563760+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'5100", 'ondisk_log_start': "37'5100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18049444, 'num_objects': 4410, 'num_object_clones': 0, 'num_object_copies': 8820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15103, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 887}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:01:55.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:01:55.997 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:55.997 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31774/136774 objects degraded (23.231%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":515,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":68387,"data_bytes":280031327,"bytes_used":1675415552,"bytes_avail":288234876928,"bytes_total":289910292480,"degraded_objects":31774,"degraded_total":136774,"degraded_ratio":0.23231023440127513,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":410},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:01:56.036 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:01:56.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:01:56.820 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:01:56.836 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:01:56.862 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-18T12:01:55.215412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280031327,"num_objects":68387,"num_object_clones":0,"num_object_copies":136774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226832,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10820,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1636148,"kb_used_data":717640,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281479372,"statfs":{"total":289910292480,"available":288234876928,"internally_reserved":0,"allocated":734863360,"data_stored":734069056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,8,10,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18038784,"num_objects":-4404,"num_object_clones":0,"num_object_copies":-8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4404,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.733982"},"pg_stats":[{"pgid":"2.f","version":"43'15298","reported_seq":15373,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:53.592828+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:53.592828+0000","last_peered":"2024-09-18T12:01:53.592828+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:53.592828+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17358848,"num_objects":4238,"num_object_clones":0,"num_object_copies":8476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15298,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":639}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15152","reported_seq":15314,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:53.431807+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:53.431807+0000","last_peered":"2024-09-18T12:01:53.431807+0000","last_clean":"2024-09-18T12:01:53.431807+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:53.431807+0000","last_undegraded":"2024-09-18T12:01:53.431807+0000","last_fullsized":"2024-09-18T12:01:53.431807+0000","mapping_epoch":29,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17530880,"num_objects":4280,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12344,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15227","reported_seq":15417,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:53.422361+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:53.422361+0000","last_peered":"2024-09-18T12:01:53.422361+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:53.422361+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17735704,"num_objects":4331,"num_object_clones":0,"num_object_copies":8662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12085,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15061","reported_seq":15094,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:53.538317+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:53.538317+0000","last_peered":"2024-09-18T12:01:53.538317+0000","last_clean":"2024-09-18T12:01:53.538317+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:53.538317+0000","last_undegraded":"2024-09-18T12:01:53.538317+0000","last_fullsized":"2024-09-18T12:01:53.538317+0000","mapping_epoch":29,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17199104,"num_objects":4199,"num_object_clones":0,"num_object_copies":8398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15061,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15136","reported_seq":15331,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:53.527450+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:53.527450+0000","last_peered":"2024-09-18T12:01:53.527450+0000","last_clean":"2024-09-18T12:01:53.527450+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:53.527450+0000","last_undegraded":"2024-09-18T12:01:53.527450+0000","last_fullsized":"2024-09-18T12:01:53.527450+0000","mapping_epoch":29,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17657856,"num_objects":4311,"num_object_clones":0,"num_object_copies":8622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12301,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15085","reported_seq":15170,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:53.583648+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:53.583648+0000","last_peered":"2024-09-18T12:01:53.583648+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:53.583648+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17769682,"num_objects":4340,"num_object_clones":0,"num_object_copies":8680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15083,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15122","reported_seq":15210,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:53.564823+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:53.564823+0000","last_peered":"2024-09-18T12:01:53.564823+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:53.564823+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17409234,"num_objects":4252,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15120,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15342","reported_seq":15570,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:53.574308+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:53.574308+0000","last_peered":"2024-09-18T12:01:53.574308+0000","last_clean":"2024-09-18T12:01:53.574308+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:53.574308+0000","last_undegraded":"2024-09-18T12:01:53.574308+0000","last_fullsized":"2024-09-18T12:01:53.574308+0000","mapping_epoch":29,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17448960,"num_objects":4260,"num_object_clones":0,"num_object_copies":8520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11659,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15107","reported_seq":22239,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:54.691175+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:54.691175+0000","last_peered":"2024-09-18T12:01:54.691175+0000","last_clean":"2024-09-18T12:01:54.691175+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:54.691175+0000","last_undegraded":"2024-09-18T12:01:54.691175+0000","last_fullsized":"2024-09-18T12:01:54.691175+0000","mapping_epoch":38,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17605842,"num_objects":4300,"num_object_clones":0,"num_object_copies":8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15105,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15157","reported_seq":15630,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:53.589067+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:01:53.589067+0000","last_peered":"2024-09-18T12:01:53.589067+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:53.589067+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17568978,"num_objects":4291,"num_object_clones":0,"num_object_copies":8582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15155,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":952}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15219","reported_seq":15272,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:53.519876+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:53.519876+0000","last_peered":"2024-09-18T12:01:53.519876+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:53.519876+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17384658,"num_objects":4246,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15217,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15219","reported_seq":22632,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:01:53.517887+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:01:53.517887+0000","last_peered":"2024-09-18T12:01:53.517887+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:01:53.517887+0000","last_undegraded":"2024-09-18T12:01:53.517887+0000","last_fullsized":"2024-09-18T12:01:53.517887+0000","mapping_epoch":42,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17526784,"num_objects":4279,"num_object_clones":0,"num_object_copies":8558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11699,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":4279}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15181","reported_seq":15267,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:53.511864+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:53.511864+0000","last_peered":"2024-09-18T12:01:53.511864+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:53.511864+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17805929,"num_objects":4348,"num_object_clones":0,"num_object_copies":8696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15180,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15284","reported_seq":22434,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:54.678765+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:54.678765+0000","last_peered":"2024-09-18T12:01:54.678765+0000","last_clean":"2024-09-18T12:01:54.678765+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:54.678765+0000","last_undegraded":"2024-09-18T12:01:54.678765+0000","last_fullsized":"2024-09-18T12:01:54.678765+0000","mapping_epoch":40,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17360699,"num_objects":4241,"num_object_clones":0,"num_object_copies":8482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15281,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15015","reported_seq":15100,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:53.562849+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:53.562849+0000","last_peered":"2024-09-18T12:01:53.562849+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:53.562849+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5000","ondisk_log_start":"37'5000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17134802,"num_objects":4185,"num_object_clones":0,"num_object_copies":8370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15013,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15232","reported_seq":15321,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:53.498324+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:53.498324+0000","last_peered":"2024-09-18T12:01:53.498324+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:53.498324+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17533348,"num_objects":4284,"num_object_clones":0,"num_object_copies":8568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15229,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":280031308,"num_objects":68385,"num_object_clones":0,"num_object_copies":136770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68385,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226830,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10819,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":733290496,"data_stored":733226688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160737,"ondisk_log_size":160737,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051680,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561212,"kb_used_data":250864,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93810628,"statfs":{"total":96636764160,"available":96062083072,"internally_reserved":0,"allocated":256884736,"data_stored":256636101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":1.7849999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":1.7090000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":2.2210000000000001},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":1.8200000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855443,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510892,"kb_used_data":204704,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93860948,"statfs":{"total":96636764160,"available":96113610752,"internally_reserved":0,"allocated":209616896,"data_stored":209319412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.373},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.23499999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.42099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564044,"kb_used_data":262072,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93807796,"statfs":{"total":96636764160,"available":96059183104,"internally_reserved":0,"allocated":268361728,"data_stored":268113543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":5.5590000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":2.5939999999999999},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":267829248,"data_stored":267825769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208977920,"data_stored":208925142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256483328,"data_stored":256475777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:01:59.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:00.753 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:00.757 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:00.772 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-18T12:01:59.216325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":272261215,"num_objects":66490,"num_object_clones":0,"num_object_copies":132980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66490,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":228729,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10873,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160737,"ondisk_log_size":160737,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1617204,"kb_used_data":707032,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281498316,"statfs":{"total":289910292480,"available":288254275584,"internally_reserved":0,"allocated":724000768,"data_stored":723206464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,5,4,8,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24797184,"num_objects":-6054,"num_object_clones":0,"num_object_copies":-12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6054,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":239,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010323"},"pg_stats":[{"pgid":"2.f","version":"43'15438","reported_seq":15513,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.561639+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:58.561639+0000","last_peered":"2024-09-18T12:01:58.561639+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:58.561639+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16785408,"num_objects":4098,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15438,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":499}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15287","reported_seq":15449,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:58.571400+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:58.571400+0000","last_peered":"2024-09-18T12:01:58.571400+0000","last_clean":"2024-09-18T12:01:58.571400+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:58.571400+0000","last_undegraded":"2024-09-18T12:01:58.571400+0000","last_fullsized":"2024-09-18T12:01:58.571400+0000","mapping_epoch":29,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16977920,"num_objects":4145,"num_object_clones":0,"num_object_copies":8290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12479,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15351","reported_seq":15541,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.578497+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:58.578497+0000","last_peered":"2024-09-18T12:01:58.578497+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:58.578497+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17227800,"num_objects":4207,"num_object_clones":0,"num_object_copies":8414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12209,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":671}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15211","reported_seq":15239,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:58.616326+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:58.616326+0000","last_peered":"2024-09-18T12:01:58.616326+0000","last_clean":"2024-09-18T12:01:58.616326+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:58.616326+0000","last_undegraded":"2024-09-18T12:01:58.616326+0000","last_fullsized":"2024-09-18T12:01:58.616326+0000","mapping_epoch":29,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16584704,"num_objects":4049,"num_object_clones":0,"num_object_copies":8098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15211,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15258","reported_seq":15453,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:58.550869+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:58.550869+0000","last_peered":"2024-09-18T12:01:58.550869+0000","last_clean":"2024-09-18T12:01:58.550869+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:58.550869+0000","last_undegraded":"2024-09-18T12:01:58.550869+0000","last_fullsized":"2024-09-18T12:01:58.550869+0000","mapping_epoch":29,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17158144,"num_objects":4189,"num_object_clones":0,"num_object_copies":8378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12423,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15225","reported_seq":15310,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.534209+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:58.534209+0000","last_peered":"2024-09-18T12:01:58.534209+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:58.534209+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17196242,"num_objects":4200,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15223,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15275","reported_seq":15366,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.595435+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:58.595435+0000","last_peered":"2024-09-18T12:01:58.595435+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:58.595435+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782546,"num_objects":4099,"num_object_clones":0,"num_object_copies":8198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15273,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":540}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15454","reported_seq":15682,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:58.607752+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:58.607752+0000","last_peered":"2024-09-18T12:01:58.607752+0000","last_clean":"2024-09-18T12:01:58.607752+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:58.607752+0000","last_undegraded":"2024-09-18T12:01:58.607752+0000","last_fullsized":"2024-09-18T12:01:58.607752+0000","mapping_epoch":29,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16990208,"num_objects":4148,"num_object_clones":0,"num_object_copies":8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15107","reported_seq":22239,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:54.691175+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:54.691175+0000","last_peered":"2024-09-18T12:01:54.691175+0000","last_clean":"2024-09-18T12:01:54.691175+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:54.691175+0000","last_undegraded":"2024-09-18T12:01:54.691175+0000","last_fullsized":"2024-09-18T12:01:54.691175+0000","mapping_epoch":38,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17605842,"num_objects":4300,"num_object_clones":0,"num_object_copies":8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15105,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15328","reported_seq":15907,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.611941+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:01:58.611941+0000","last_peered":"2024-09-18T12:01:58.611941+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:58.611941+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16868562,"num_objects":4120,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15326,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":834}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15336","reported_seq":15395,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.594182+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:58.594182+0000","last_peered":"2024-09-18T12:01:58.594182+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:58.594182+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16905426,"num_objects":4129,"num_object_clones":0,"num_object_copies":8258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15334,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":579}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15337","reported_seq":22750,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:01:58.533638+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:01:58.533638+0000","last_peered":"2024-09-18T12:01:58.533638+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:01:58.533638+0000","last_undegraded":"2024-09-18T12:01:58.533638+0000","last_fullsized":"2024-09-18T12:01:58.533638+0000","mapping_epoch":42,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17043456,"num_objects":4161,"num_object_clones":0,"num_object_copies":8322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11817,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":4161}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15308","reported_seq":15394,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.511791+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:58.511791+0000","last_peered":"2024-09-18T12:01:58.511791+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:58.511791+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17285737,"num_objects":4221,"num_object_clones":0,"num_object_copies":8442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15307,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15284","reported_seq":22434,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:54.678765+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:54.678765+0000","last_peered":"2024-09-18T12:01:54.678765+0000","last_clean":"2024-09-18T12:01:54.678765+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:54.678765+0000","last_undegraded":"2024-09-18T12:01:54.678765+0000","last_fullsized":"2024-09-18T12:01:54.678765+0000","mapping_epoch":40,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17360699,"num_objects":4241,"num_object_clones":0,"num_object_copies":8482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15281,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15163","reported_seq":15248,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.591999+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:58.591999+0000","last_peered":"2024-09-18T12:01:58.591999+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:58.591999+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16528594,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15161,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15372","reported_seq":15461,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.547994+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:58.547994+0000","last_peered":"2024-09-18T12:01:58.547994+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:58.547994+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16959908,"num_objects":4144,"num_object_clones":0,"num_object_copies":8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15369,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":621}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":272261196,"num_objects":66488,"num_object_clones":0,"num_object_copies":132976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":228727,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10872,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":722321408,"data_stored":722257600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160734,"ondisk_log_size":160734,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051681,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544812,"kb_used_data":246948,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93827028,"statfs":{"total":96636764160,"available":96078876672,"internally_reserved":0,"allocated":252874752,"data_stored":252626117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.249},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":9.5169999999999995}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.35199999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855443,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510892,"kb_used_data":204704,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93860948,"statfs":{"total":96636764160,"available":96113610752,"internally_reserved":0,"allocated":209616896,"data_stored":209319412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.373},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.23499999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.42099999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561500,"kb_used_data":255380,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93810340,"statfs":{"total":96636764160,"available":96061788160,"internally_reserved":0,"allocated":261509120,"data_stored":261260935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,1,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.222},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":3.96},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.34799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":260968448,"data_stored":260964969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208977920,"data_stored":208925142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":252375040,"data_stored":252367489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:00.774 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:01.322 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:01.328 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:01.348 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-09-18T12:02:01.216782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":271052895,"num_objects":66195,"num_object_clones":0,"num_object_copies":132390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66195,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229024,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10873,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160732,"ondisk_log_size":160732,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1614580,"kb_used_data":700252,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281500940,"statfs":{"total":289910292480,"available":288256962560,"internally_reserved":0,"allocated":717058048,"data_stored":716263744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,7,2,9,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19095552,"num_objects":-4662,"num_object_clones":0,"num_object_copies":-9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4662,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010381"},"pg_stats":[{"pgid":"2.f","version":"43'15438","reported_seq":15513,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.561639+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:58.561639+0000","last_peered":"2024-09-18T12:01:58.561639+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:58.561639+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16785408,"num_objects":4098,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15438,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":499}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15287","reported_seq":15449,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:58.571400+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:58.571400+0000","last_peered":"2024-09-18T12:01:58.571400+0000","last_clean":"2024-09-18T12:01:58.571400+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:58.571400+0000","last_undegraded":"2024-09-18T12:01:58.571400+0000","last_fullsized":"2024-09-18T12:01:58.571400+0000","mapping_epoch":29,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16977920,"num_objects":4145,"num_object_clones":0,"num_object_copies":8290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12479,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15351","reported_seq":15541,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.578497+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:58.578497+0000","last_peered":"2024-09-18T12:01:58.578497+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:58.578497+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17227800,"num_objects":4207,"num_object_clones":0,"num_object_copies":8414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12209,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":671}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15211","reported_seq":15239,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:58.616326+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:58.616326+0000","last_peered":"2024-09-18T12:01:58.616326+0000","last_clean":"2024-09-18T12:01:58.616326+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:58.616326+0000","last_undegraded":"2024-09-18T12:01:58.616326+0000","last_fullsized":"2024-09-18T12:01:58.616326+0000","mapping_epoch":29,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16584704,"num_objects":4049,"num_object_clones":0,"num_object_copies":8098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15211,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15258","reported_seq":15453,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:58.550869+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:58.550869+0000","last_peered":"2024-09-18T12:01:58.550869+0000","last_clean":"2024-09-18T12:01:58.550869+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:58.550869+0000","last_undegraded":"2024-09-18T12:01:58.550869+0000","last_fullsized":"2024-09-18T12:01:58.550869+0000","mapping_epoch":29,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17158144,"num_objects":4189,"num_object_clones":0,"num_object_copies":8378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12423,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15225","reported_seq":15310,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.534209+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:58.534209+0000","last_peered":"2024-09-18T12:01:58.534209+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:58.534209+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17196242,"num_objects":4200,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15223,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15275","reported_seq":15366,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.595435+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:58.595435+0000","last_peered":"2024-09-18T12:01:58.595435+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:58.595435+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782546,"num_objects":4099,"num_object_clones":0,"num_object_copies":8198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15273,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":540}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15454","reported_seq":15682,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:58.607752+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:58.607752+0000","last_peered":"2024-09-18T12:01:58.607752+0000","last_clean":"2024-09-18T12:01:58.607752+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:58.607752+0000","last_undegraded":"2024-09-18T12:01:58.607752+0000","last_fullsized":"2024-09-18T12:01:58.607752+0000","mapping_epoch":29,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16990208,"num_objects":4148,"num_object_clones":0,"num_object_copies":8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15244","reported_seq":22376,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:59.745784+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:59.745784+0000","last_peered":"2024-09-18T12:01:59.745784+0000","last_clean":"2024-09-18T12:01:59.745784+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:59.745784+0000","last_undegraded":"2024-09-18T12:01:59.745784+0000","last_fullsized":"2024-09-18T12:01:59.745784+0000","mapping_epoch":38,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044690,"num_objects":4163,"num_object_clones":0,"num_object_copies":8326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15242,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15328","reported_seq":15907,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.611941+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:01:58.611941+0000","last_peered":"2024-09-18T12:01:58.611941+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:58.611941+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16868562,"num_objects":4120,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15326,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":834}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15336","reported_seq":15395,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.594182+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:58.594182+0000","last_peered":"2024-09-18T12:01:58.594182+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:58.594182+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16905426,"num_objects":4129,"num_object_clones":0,"num_object_copies":8258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15334,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":579}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15337","reported_seq":22750,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:01:58.533638+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:01:58.533638+0000","last_peered":"2024-09-18T12:01:58.533638+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:01:58.533638+0000","last_undegraded":"2024-09-18T12:01:58.533638+0000","last_fullsized":"2024-09-18T12:01:58.533638+0000","mapping_epoch":42,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17043456,"num_objects":4161,"num_object_clones":0,"num_object_copies":8322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11817,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":4161}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15308","reported_seq":15394,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.511791+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:58.511791+0000","last_peered":"2024-09-18T12:01:58.511791+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:58.511791+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17285737,"num_objects":4221,"num_object_clones":0,"num_object_copies":8442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15307,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15442","reported_seq":22592,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:59.695768+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:59.695768+0000","last_peered":"2024-09-18T12:01:59.695768+0000","last_clean":"2024-09-18T12:01:59.695768+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:59.695768+0000","last_undegraded":"2024-09-18T12:01:59.695768+0000","last_fullsized":"2024-09-18T12:01:59.695768+0000","mapping_epoch":40,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16713531,"num_objects":4083,"num_object_clones":0,"num_object_copies":8166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15439,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15163","reported_seq":15248,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.591999+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:58.591999+0000","last_peered":"2024-09-18T12:01:58.591999+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:58.591999+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16528594,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15161,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15372","reported_seq":15461,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.547994+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:58.547994+0000","last_peered":"2024-09-18T12:01:58.547994+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:58.547994+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16959908,"num_objects":4144,"num_object_clones":0,"num_object_copies":8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15369,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":621}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":271052876,"num_objects":66193,"num_object_clones":0,"num_object_copies":132386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66193,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229022,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10872,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715345920,"data_stored":715282112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160729,"ondisk_log_size":160729,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051681,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544812,"kb_used_data":246948,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93827028,"statfs":{"total":96636764160,"available":96078876672,"internally_reserved":0,"allocated":252874752,"data_stored":252626117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.249},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":9.5169999999999995}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.35199999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855444,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508268,"kb_used_data":197924,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93863572,"statfs":{"total":96636764160,"available":96116297728,"internally_reserved":0,"allocated":202674176,"data_stored":202376692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,0,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":3.4830000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.219},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.29599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561500,"kb_used_data":255380,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93810340,"statfs":{"total":96636764160,"available":96061788160,"internally_reserved":0,"allocated":261509120,"data_stored":261260935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,1,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.222},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":3.96},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.34799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":260968448,"data_stored":260964969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":202002432,"data_stored":201949654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":252375040,"data_stored":252367489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:01.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:02:01.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'15438", 'reported_seq': 15513, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:58.561639+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:01:58.561639+0000', 'last_peered': '2024-09-18T12:01:58.561639+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:01:58.561639+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'5400", 'ondisk_log_start': "37'5400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16785408, 'num_objects': 4098, 'num_object_clones': 0, 'num_object_copies': 8196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15438, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 499}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:01.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:02:01.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'15351", 'reported_seq': 15541, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:58.578497+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:01:58.578497+0000', 'last_peered': '2024-09-18T12:01:58.578497+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:01:58.578497+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'5300", 'ondisk_log_start': "37'5300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17227800, 'num_objects': 4207, 'num_object_clones': 0, 'num_object_copies': 8414, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4207, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12209, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 671}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:01.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:02:01.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'15225", 'reported_seq': 15310, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:58.534209+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:01:58.534209+0000', 'last_peered': '2024-09-18T12:01:58.534209+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:01:58.534209+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'5200", 'ondisk_log_start': "37'5200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17196242, 'num_objects': 4200, 'num_object_clones': 0, 'num_object_copies': 8400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15223, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 673}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:01.351 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:02:01.351 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'15275", 'reported_seq': 15366, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:58.595435+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:01:58.595435+0000', 'last_peered': '2024-09-18T12:01:58.595435+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:01:58.595435+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'5200", 'ondisk_log_start': "37'5200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16782546, 'num_objects': 4099, 'num_object_clones': 0, 'num_object_copies': 8198, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15273, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 540}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:01.351 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:02:01.351 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'15328", 'reported_seq': 15907, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:58.611941+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:01:58.611941+0000', 'last_peered': '2024-09-18T12:01:58.611941+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:01:58.611941+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'5300", 'ondisk_log_start': "37'5300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16868562, 'num_objects': 4120, 'num_object_clones': 0, 'num_object_copies': 8240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3286, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15326, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 250, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 834}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:01.351 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:02:01.351 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'15336", 'reported_seq': 15395, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:58.594182+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:01:58.594182+0000', 'last_peered': '2024-09-18T12:01:58.594182+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:01:58.594182+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'5300", 'ondisk_log_start': "37'5300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16905426, 'num_objects': 4129, 'num_object_clones': 0, 'num_object_copies': 8258, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4129, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15334, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 579}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:01.351 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:02:01.351 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'15337", 'reported_seq': 22750, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:01:58.533638+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:01:58.533638+0000', 'last_peered': '2024-09-18T12:01:58.533638+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:01:58.533638+0000', 'last_undegraded': '2024-09-18T12:01:58.533638+0000', 'last_fullsized': '2024-09-18T12:01:58.533638+0000', 'mapping_epoch': 42, 'log_start': "37'5300", 'ondisk_log_start': "37'5300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17043456, 'num_objects': 4161, 'num_object_clones': 0, 'num_object_copies': 8322, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11817, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 4161}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:01.351 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:02:01.351 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'15308", 'reported_seq': 15394, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:58.511791+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:01:58.511791+0000', 'last_peered': '2024-09-18T12:01:58.511791+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:01:58.511791+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'5300", 'ondisk_log_start': "37'5300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17285737, 'num_objects': 4221, 'num_object_clones': 0, 'num_object_copies': 8442, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4221, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15307, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:01.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:02:01.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'15163", 'reported_seq': 15248, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:58.591999+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:01:58.591999+0000', 'last_peered': '2024-09-18T12:01:58.591999+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:01:58.591999+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'5100", 'ondisk_log_start': "37'5100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16528594, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15161, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 444}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:01.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:02:01.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'15372", 'reported_seq': 15461, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:01:58.547994+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:01:58.547994+0000', 'last_peered': '2024-09-18T12:01:58.547994+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:01:58.547994+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'5300", 'ondisk_log_start': "37'5300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16959908, 'num_objects': 4144, 'num_object_clones': 0, 'num_object_copies': 8288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4144, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15369, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 621}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:01.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:02:02.148 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:02.149 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31721/132390 objects degraded (23.960%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":522,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":66195,"data_bytes":271052895,"bytes_used":1653329920,"bytes_avail":288256962560,"bytes_total":289910292480,"degraded_objects":31721,"degraded_total":132390,"degraded_ratio":0.23960268902485082,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":388},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:02:02.194 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:02:02.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:02.942 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:02.959 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:02.973 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-09-18T12:02:01.216782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":271052895,"num_objects":66195,"num_object_clones":0,"num_object_copies":132390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66195,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229024,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10873,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160732,"ondisk_log_size":160732,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1614580,"kb_used_data":700252,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281500940,"statfs":{"total":289910292480,"available":288256962560,"internally_reserved":0,"allocated":717058048,"data_stored":716263744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,7,2,9,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19095552,"num_objects":-4662,"num_object_clones":0,"num_object_copies":-9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4662,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010381"},"pg_stats":[{"pgid":"2.f","version":"43'15438","reported_seq":15513,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.561639+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:01:58.561639+0000","last_peered":"2024-09-18T12:01:58.561639+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:01:58.561639+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16785408,"num_objects":4098,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15438,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":499}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15287","reported_seq":15449,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:58.571400+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:01:58.571400+0000","last_peered":"2024-09-18T12:01:58.571400+0000","last_clean":"2024-09-18T12:01:58.571400+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:01:58.571400+0000","last_undegraded":"2024-09-18T12:01:58.571400+0000","last_fullsized":"2024-09-18T12:01:58.571400+0000","mapping_epoch":29,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16977920,"num_objects":4145,"num_object_clones":0,"num_object_copies":8290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12479,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15351","reported_seq":15541,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.578497+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:01:58.578497+0000","last_peered":"2024-09-18T12:01:58.578497+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:01:58.578497+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17227800,"num_objects":4207,"num_object_clones":0,"num_object_copies":8414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12209,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":671}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15211","reported_seq":15239,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:58.616326+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:01:58.616326+0000","last_peered":"2024-09-18T12:01:58.616326+0000","last_clean":"2024-09-18T12:01:58.616326+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:01:58.616326+0000","last_undegraded":"2024-09-18T12:01:58.616326+0000","last_fullsized":"2024-09-18T12:01:58.616326+0000","mapping_epoch":29,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16584704,"num_objects":4049,"num_object_clones":0,"num_object_copies":8098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15211,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15258","reported_seq":15453,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:58.550869+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:01:58.550869+0000","last_peered":"2024-09-18T12:01:58.550869+0000","last_clean":"2024-09-18T12:01:58.550869+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:01:58.550869+0000","last_undegraded":"2024-09-18T12:01:58.550869+0000","last_fullsized":"2024-09-18T12:01:58.550869+0000","mapping_epoch":29,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17158144,"num_objects":4189,"num_object_clones":0,"num_object_copies":8378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12423,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15225","reported_seq":15310,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.534209+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:01:58.534209+0000","last_peered":"2024-09-18T12:01:58.534209+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:01:58.534209+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17196242,"num_objects":4200,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15223,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15275","reported_seq":15366,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.595435+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:01:58.595435+0000","last_peered":"2024-09-18T12:01:58.595435+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:01:58.595435+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16782546,"num_objects":4099,"num_object_clones":0,"num_object_copies":8198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15273,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":540}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15454","reported_seq":15682,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:58.607752+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:01:58.607752+0000","last_peered":"2024-09-18T12:01:58.607752+0000","last_clean":"2024-09-18T12:01:58.607752+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:01:58.607752+0000","last_undegraded":"2024-09-18T12:01:58.607752+0000","last_fullsized":"2024-09-18T12:01:58.607752+0000","mapping_epoch":29,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16990208,"num_objects":4148,"num_object_clones":0,"num_object_copies":8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15244","reported_seq":22376,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:59.745784+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:01:59.745784+0000","last_peered":"2024-09-18T12:01:59.745784+0000","last_clean":"2024-09-18T12:01:59.745784+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:01:59.745784+0000","last_undegraded":"2024-09-18T12:01:59.745784+0000","last_fullsized":"2024-09-18T12:01:59.745784+0000","mapping_epoch":38,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044690,"num_objects":4163,"num_object_clones":0,"num_object_copies":8326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15242,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15328","reported_seq":15907,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.611941+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:01:58.611941+0000","last_peered":"2024-09-18T12:01:58.611941+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:01:58.611941+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16868562,"num_objects":4120,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15326,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":834}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15336","reported_seq":15395,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.594182+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:01:58.594182+0000","last_peered":"2024-09-18T12:01:58.594182+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:01:58.594182+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16905426,"num_objects":4129,"num_object_clones":0,"num_object_copies":8258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15334,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":579}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15337","reported_seq":22750,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:01:58.533638+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:01:58.533638+0000","last_peered":"2024-09-18T12:01:58.533638+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:01:58.533638+0000","last_undegraded":"2024-09-18T12:01:58.533638+0000","last_fullsized":"2024-09-18T12:01:58.533638+0000","mapping_epoch":42,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17043456,"num_objects":4161,"num_object_clones":0,"num_object_copies":8322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11817,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":4161}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15308","reported_seq":15394,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.511791+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:01:58.511791+0000","last_peered":"2024-09-18T12:01:58.511791+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:01:58.511791+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17285737,"num_objects":4221,"num_object_clones":0,"num_object_copies":8442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15307,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15442","reported_seq":22592,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:59.695768+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:01:59.695768+0000","last_peered":"2024-09-18T12:01:59.695768+0000","last_clean":"2024-09-18T12:01:59.695768+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:01:59.695768+0000","last_undegraded":"2024-09-18T12:01:59.695768+0000","last_fullsized":"2024-09-18T12:01:59.695768+0000","mapping_epoch":40,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16713531,"num_objects":4083,"num_object_clones":0,"num_object_copies":8166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15439,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15163","reported_seq":15248,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.591999+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:01:58.591999+0000","last_peered":"2024-09-18T12:01:58.591999+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:01:58.591999+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5100","ondisk_log_start":"37'5100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16528594,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15161,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15372","reported_seq":15461,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:01:58.547994+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:01:58.547994+0000","last_peered":"2024-09-18T12:01:58.547994+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:01:58.547994+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16959908,"num_objects":4144,"num_object_clones":0,"num_object_copies":8288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15369,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":621}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":271052876,"num_objects":66193,"num_object_clones":0,"num_object_copies":132386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66193,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229022,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10872,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715345920,"data_stored":715282112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160729,"ondisk_log_size":160729,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051681,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544812,"kb_used_data":246948,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93827028,"statfs":{"total":96636764160,"available":96078876672,"internally_reserved":0,"allocated":252874752,"data_stored":252626117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.249},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":9.5169999999999995}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.35199999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855444,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508268,"kb_used_data":197924,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93863572,"statfs":{"total":96636764160,"available":96116297728,"internally_reserved":0,"allocated":202674176,"data_stored":202376692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,0,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":3.4830000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.219},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.29599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561500,"kb_used_data":255380,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93810340,"statfs":{"total":96636764160,"available":96061788160,"internally_reserved":0,"allocated":261509120,"data_stored":261260935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,1,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.222},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":3.96},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.34799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":260968448,"data_stored":260964969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":202002432,"data_stored":201949654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":252375040,"data_stored":252367489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:05.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:06.630 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:06.640 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:06.657 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-18T12:02:05.217615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262287455,"num_objects":64055,"num_object_clones":0,"num_object_copies":128110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64055,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231164,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10930,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160872,"ondisk_log_size":160872,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1597516,"kb_used_data":683176,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281518004,"statfs":{"total":289910292480,"available":288274436096,"internally_reserved":0,"allocated":699572224,"data_stored":698777920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,8,5,9,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26755072,"num_objects":-6532,"num_object_clones":0,"num_object_copies":-13064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6532,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006227"},"pg_stats":[{"pgid":"2.f","version":"43'15554","reported_seq":15629,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.561072+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:03.561072+0000","last_peered":"2024-09-18T12:02:03.561072+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:03.561072+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16310272,"num_objects":3982,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15554,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":383}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15410","reported_seq":15572,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:03.396262+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:03.396262+0000","last_peered":"2024-09-18T12:02:03.396262+0000","last_clean":"2024-09-18T12:02:03.396262+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:03.396262+0000","last_undegraded":"2024-09-18T12:02:03.396262+0000","last_fullsized":"2024-09-18T12:02:03.396262+0000","mapping_epoch":29,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16474112,"num_objects":4022,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12602,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15492","reported_seq":15682,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.613901+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:03.613901+0000","last_peered":"2024-09-18T12:02:03.613901+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:03.613901+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16650264,"num_objects":4066,"num_object_clones":0,"num_object_copies":8132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12350,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":530}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15330","reported_seq":15363,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:03.466781+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:03.466781+0000","last_peered":"2024-09-18T12:02:03.466781+0000","last_clean":"2024-09-18T12:02:03.466781+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:03.466781+0000","last_undegraded":"2024-09-18T12:02:03.466781+0000","last_fullsized":"2024-09-18T12:02:03.466781+0000","mapping_epoch":29,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16097280,"num_objects":3930,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15330,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15406","reported_seq":15601,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:03.557131+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:03.557131+0000","last_peered":"2024-09-18T12:02:03.557131+0000","last_clean":"2024-09-18T12:02:03.557131+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:03.557131+0000","last_undegraded":"2024-09-18T12:02:03.557131+0000","last_fullsized":"2024-09-18T12:02:03.557131+0000","mapping_epoch":29,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16551936,"num_objects":4041,"num_object_clones":0,"num_object_copies":8082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15371","reported_seq":15455,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.563690+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:03.563690+0000","last_peered":"2024-09-18T12:02:03.563690+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:03.563690+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16598226,"num_objects":4054,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15369,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15405","reported_seq":15496,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.577463+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:03.577463+0000","last_peered":"2024-09-18T12:02:03.577463+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:03.577463+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16250066,"num_objects":3969,"num_object_clones":0,"num_object_copies":7938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15403,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15566","reported_seq":15795,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:03.484367+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:03.484367+0000","last_peered":"2024-09-18T12:02:03.484367+0000","last_clean":"2024-09-18T12:02:03.484367+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:03.484367+0000","last_undegraded":"2024-09-18T12:02:03.484367+0000","last_fullsized":"2024-09-18T12:02:03.484367+0000","mapping_epoch":29,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16531456,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11883,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15394","reported_seq":22526,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:04.872414+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:04.872414+0000","last_peered":"2024-09-18T12:02:04.872414+0000","last_clean":"2024-09-18T12:02:04.872414+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:04.872414+0000","last_undegraded":"2024-09-18T12:02:04.872414+0000","last_fullsized":"2024-09-18T12:02:04.872414+0000","mapping_epoch":38,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16430290,"num_objects":4013,"num_object_clones":0,"num_object_copies":8026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15392,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15471","reported_seq":16163,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.632262+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:03.632262+0000","last_peered":"2024-09-18T12:02:03.632262+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:03.632262+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16282834,"num_objects":3977,"num_object_clones":0,"num_object_copies":7954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15469,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15474","reported_seq":15534,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.588268+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:03.588268+0000","last_peered":"2024-09-18T12:02:03.588268+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:03.588268+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16340178,"num_objects":3991,"num_object_clones":0,"num_object_copies":7982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15472,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":441}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15479","reported_seq":22892,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:03.608055+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:03.608055+0000","last_peered":"2024-09-18T12:02:03.608055+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:03.608055+0000","last_undegraded":"2024-09-18T12:02:03.608055+0000","last_fullsized":"2024-09-18T12:02:03.608055+0000","mapping_epoch":42,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16461824,"num_objects":4019,"num_object_clones":0,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11959,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":4019}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15424","reported_seq":15509,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.586384+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:03.586384+0000","last_peered":"2024-09-18T12:02:03.586384+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:03.586384+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16810601,"num_objects":4105,"num_object_clones":0,"num_object_copies":8210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15423,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15579","reported_seq":22729,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:04.836043+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:04.836043+0000","last_peered":"2024-09-18T12:02:04.836043+0000","last_clean":"2024-09-18T12:02:04.836043+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:04.836043+0000","last_undegraded":"2024-09-18T12:02:04.836043+0000","last_fullsized":"2024-09-18T12:02:04.836043+0000","mapping_epoch":40,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16152379,"num_objects":3946,"num_object_clones":0,"num_object_copies":7892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15576,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15295","reported_seq":15380,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.508388+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:03.508388+0000","last_peered":"2024-09-18T12:02:03.508388+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:03.508388+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15987922,"num_objects":3905,"num_object_clones":0,"num_object_copies":7810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15293,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15519","reported_seq":15608,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.585264+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:03.585264+0000","last_peered":"2024-09-18T12:02:03.585264+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:03.585264+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16357796,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15516,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":262287436,"num_objects":64053,"num_object_clones":0,"num_object_copies":128106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231162,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10929,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":697663488,"data_stored":697599680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160869,"ondisk_log_size":160869,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051682,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544700,"kb_used_data":242676,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93827140,"statfs":{"total":96636764160,"available":96078991360,"internally_reserved":0,"allocated":248500224,"data_stored":248251589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.626},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.438},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.66300000000000003}]}]},{"osd":1,"up_from":35,"seq":150323855445,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493564,"kb_used_data":191536,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93878276,"statfs":{"total":96636764160,"available":96131354624,"internally_reserved":0,"allocated":196132864,"data_stored":195835380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,2,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.311},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.34899999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559252,"kb_used_data":248964,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93812588,"statfs":{"total":96636764160,"available":96064090112,"internally_reserved":0,"allocated":254939136,"data_stored":254690951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,2,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":3.508},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":10.525},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":254443520,"data_stored":254440041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195211264,"data_stored":195158486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":248008704,"data_stored":248001153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:06.659 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:07.350 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:07.374 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:07.411 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-18T12:02:07.218029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262287455,"num_objects":64055,"num_object_clones":0,"num_object_copies":128110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64055,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231164,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10930,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160872,"ondisk_log_size":160872,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1597516,"kb_used_data":683176,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281518004,"statfs":{"total":289910292480,"available":288274436096,"internally_reserved":0,"allocated":699572224,"data_stored":698777920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,8,5,9,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17743872,"num_objects":-4332,"num_object_clones":0,"num_object_copies":-8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4332,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002616"},"pg_stats":[{"pgid":"2.f","version":"43'15554","reported_seq":15629,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.561072+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:03.561072+0000","last_peered":"2024-09-18T12:02:03.561072+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:03.561072+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16310272,"num_objects":3982,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15554,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":383}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15410","reported_seq":15572,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:03.396262+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:03.396262+0000","last_peered":"2024-09-18T12:02:03.396262+0000","last_clean":"2024-09-18T12:02:03.396262+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:03.396262+0000","last_undegraded":"2024-09-18T12:02:03.396262+0000","last_fullsized":"2024-09-18T12:02:03.396262+0000","mapping_epoch":29,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16474112,"num_objects":4022,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12602,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15492","reported_seq":15682,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.613901+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:03.613901+0000","last_peered":"2024-09-18T12:02:03.613901+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:03.613901+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16650264,"num_objects":4066,"num_object_clones":0,"num_object_copies":8132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12350,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":530}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15330","reported_seq":15363,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:03.466781+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:03.466781+0000","last_peered":"2024-09-18T12:02:03.466781+0000","last_clean":"2024-09-18T12:02:03.466781+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:03.466781+0000","last_undegraded":"2024-09-18T12:02:03.466781+0000","last_fullsized":"2024-09-18T12:02:03.466781+0000","mapping_epoch":29,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16097280,"num_objects":3930,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15330,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15406","reported_seq":15601,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:03.557131+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:03.557131+0000","last_peered":"2024-09-18T12:02:03.557131+0000","last_clean":"2024-09-18T12:02:03.557131+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:03.557131+0000","last_undegraded":"2024-09-18T12:02:03.557131+0000","last_fullsized":"2024-09-18T12:02:03.557131+0000","mapping_epoch":29,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16551936,"num_objects":4041,"num_object_clones":0,"num_object_copies":8082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15371","reported_seq":15455,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.563690+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:03.563690+0000","last_peered":"2024-09-18T12:02:03.563690+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:03.563690+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16598226,"num_objects":4054,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15369,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15405","reported_seq":15496,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.577463+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:03.577463+0000","last_peered":"2024-09-18T12:02:03.577463+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:03.577463+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16250066,"num_objects":3969,"num_object_clones":0,"num_object_copies":7938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15403,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15566","reported_seq":15795,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:03.484367+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:03.484367+0000","last_peered":"2024-09-18T12:02:03.484367+0000","last_clean":"2024-09-18T12:02:03.484367+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:03.484367+0000","last_undegraded":"2024-09-18T12:02:03.484367+0000","last_fullsized":"2024-09-18T12:02:03.484367+0000","mapping_epoch":29,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16531456,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11883,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15394","reported_seq":22526,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:04.872414+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:04.872414+0000","last_peered":"2024-09-18T12:02:04.872414+0000","last_clean":"2024-09-18T12:02:04.872414+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:04.872414+0000","last_undegraded":"2024-09-18T12:02:04.872414+0000","last_fullsized":"2024-09-18T12:02:04.872414+0000","mapping_epoch":38,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16430290,"num_objects":4013,"num_object_clones":0,"num_object_copies":8026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15392,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15471","reported_seq":16163,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.632262+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:03.632262+0000","last_peered":"2024-09-18T12:02:03.632262+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:03.632262+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16282834,"num_objects":3977,"num_object_clones":0,"num_object_copies":7954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15469,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15474","reported_seq":15534,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.588268+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:03.588268+0000","last_peered":"2024-09-18T12:02:03.588268+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:03.588268+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16340178,"num_objects":3991,"num_object_clones":0,"num_object_copies":7982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15472,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":441}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15479","reported_seq":22892,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:03.608055+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:03.608055+0000","last_peered":"2024-09-18T12:02:03.608055+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:03.608055+0000","last_undegraded":"2024-09-18T12:02:03.608055+0000","last_fullsized":"2024-09-18T12:02:03.608055+0000","mapping_epoch":42,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16461824,"num_objects":4019,"num_object_clones":0,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11959,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":4019}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15424","reported_seq":15509,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.586384+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:03.586384+0000","last_peered":"2024-09-18T12:02:03.586384+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:03.586384+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16810601,"num_objects":4105,"num_object_clones":0,"num_object_copies":8210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15423,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15579","reported_seq":22729,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:04.836043+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:04.836043+0000","last_peered":"2024-09-18T12:02:04.836043+0000","last_clean":"2024-09-18T12:02:04.836043+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:04.836043+0000","last_undegraded":"2024-09-18T12:02:04.836043+0000","last_fullsized":"2024-09-18T12:02:04.836043+0000","mapping_epoch":40,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16152379,"num_objects":3946,"num_object_clones":0,"num_object_copies":7892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15576,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15295","reported_seq":15380,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.508388+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:03.508388+0000","last_peered":"2024-09-18T12:02:03.508388+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:03.508388+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15987922,"num_objects":3905,"num_object_clones":0,"num_object_copies":7810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15293,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15519","reported_seq":15608,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.585264+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:03.585264+0000","last_peered":"2024-09-18T12:02:03.585264+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:03.585264+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16357796,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15516,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":262287436,"num_objects":64053,"num_object_clones":0,"num_object_copies":128106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231162,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10929,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":697663488,"data_stored":697599680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160869,"ondisk_log_size":160869,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051682,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544700,"kb_used_data":242676,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93827140,"statfs":{"total":96636764160,"available":96078991360,"internally_reserved":0,"allocated":248500224,"data_stored":248251589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.626},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.438},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.66300000000000003}]}]},{"osd":1,"up_from":35,"seq":150323855445,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493564,"kb_used_data":191536,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93878276,"statfs":{"total":96636764160,"available":96131354624,"internally_reserved":0,"allocated":196132864,"data_stored":195835380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,2,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.311},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.34899999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559252,"kb_used_data":248964,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93812588,"statfs":{"total":96636764160,"available":96064090112,"internally_reserved":0,"allocated":254939136,"data_stored":254690951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,2,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":3.508},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":10.525},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":254443520,"data_stored":254440041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195211264,"data_stored":195158486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":248008704,"data_stored":248001153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:07.413 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:02:07.413 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'15554", 'reported_seq': 15629, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:03.561072+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:02:03.561072+0000', 'last_peered': '2024-09-18T12:02:03.561072+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:02:03.561072+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'5500", 'ondisk_log_start': "37'5500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16310272, 'num_objects': 3982, 'num_object_clones': 0, 'num_object_copies': 7964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15554, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 383}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:07.413 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:02:07.413 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'15492", 'reported_seq': 15682, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:03.613901+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:02:03.613901+0000', 'last_peered': '2024-09-18T12:02:03.613901+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:02:03.613901+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'5400", 'ondisk_log_start': "37'5400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16650264, 'num_objects': 4066, 'num_object_clones': 0, 'num_object_copies': 8132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4066, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12350, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 530}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:07.413 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:02:07.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'15371", 'reported_seq': 15455, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:03.563690+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:02:03.563690+0000', 'last_peered': '2024-09-18T12:02:03.563690+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:02:03.563690+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'5300", 'ondisk_log_start': "37'5300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16598226, 'num_objects': 4054, 'num_object_clones': 0, 'num_object_copies': 8108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15369, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:07.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:02:07.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'15405", 'reported_seq': 15496, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:03.577463+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:02:03.577463+0000', 'last_peered': '2024-09-18T12:02:03.577463+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:02:03.577463+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'5400", 'ondisk_log_start': "37'5400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16250066, 'num_objects': 3969, 'num_object_clones': 0, 'num_object_copies': 7938, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15403, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 410}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:07.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:02:07.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'15471", 'reported_seq': 16163, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:03.632262+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:02:03.632262+0000', 'last_peered': '2024-09-18T12:02:03.632262+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:02:03.632262+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'5400", 'ondisk_log_start': "37'5400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16282834, 'num_objects': 3977, 'num_object_clones': 0, 'num_object_copies': 7954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15469, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 307, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 748}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:07.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:02:07.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'15474", 'reported_seq': 15534, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:03.588268+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:02:03.588268+0000', 'last_peered': '2024-09-18T12:02:03.588268+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:02:03.588268+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'5400", 'ondisk_log_start': "37'5400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16340178, 'num_objects': 3991, 'num_object_clones': 0, 'num_object_copies': 7982, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15472, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 441}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:07.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:02:07.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'15479", 'reported_seq': 22892, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:02:03.608055+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:02:03.608055+0000', 'last_peered': '2024-09-18T12:02:03.608055+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:02:03.608055+0000', 'last_undegraded': '2024-09-18T12:02:03.608055+0000', 'last_fullsized': '2024-09-18T12:02:03.608055+0000', 'mapping_epoch': 42, 'log_start': "37'5400", 'ondisk_log_start': "37'5400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16461824, 'num_objects': 4019, 'num_object_clones': 0, 'num_object_copies': 8038, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11959, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 4019}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:07.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:02:07.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'15424", 'reported_seq': 15509, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:03.586384+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:02:03.586384+0000', 'last_peered': '2024-09-18T12:02:03.586384+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:02:03.586384+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'5400", 'ondisk_log_start': "37'5400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16810601, 'num_objects': 4105, 'num_object_clones': 0, 'num_object_copies': 8210, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4105, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15423, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 570}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:07.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:02:07.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'15295", 'reported_seq': 15380, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:03.508388+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:02:03.508388+0000', 'last_peered': '2024-09-18T12:02:03.508388+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:02:03.508388+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'5200", 'ondisk_log_start': "37'5200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15987922, 'num_objects': 3905, 'num_object_clones': 0, 'num_object_copies': 7810, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15293, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 312}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:07.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:02:07.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'15519", 'reported_seq': 15608, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:03.585264+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:02:03.585264+0000', 'last_peered': '2024-09-18T12:02:03.585264+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:02:03.585264+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'5500", 'ondisk_log_start': "37'5500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16357796, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15516, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 474}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:07.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:02:08.355 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:08.356 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31664/128110 objects degraded (24.716%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":528,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":64055,"data_bytes":262287455,"bytes_used":1635856384,"bytes_avail":288274436096,"bytes_total":289910292480,"degraded_objects":31664,"degraded_total":128110,"degraded_ratio":0.24716259464522675,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":360},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:02:08.410 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:02:08.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:09.148 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:09.173 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:09.178 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-18T12:02:07.218029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262287455,"num_objects":64055,"num_object_clones":0,"num_object_copies":128110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64055,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231164,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10930,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160872,"ondisk_log_size":160872,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1597516,"kb_used_data":683176,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281518004,"statfs":{"total":289910292480,"available":288274436096,"internally_reserved":0,"allocated":699572224,"data_stored":698777920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,8,5,9,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17743872,"num_objects":-4332,"num_object_clones":0,"num_object_copies":-8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4332,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002616"},"pg_stats":[{"pgid":"2.f","version":"43'15554","reported_seq":15629,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.561072+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:03.561072+0000","last_peered":"2024-09-18T12:02:03.561072+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:03.561072+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16310272,"num_objects":3982,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15554,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":383}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15410","reported_seq":15572,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:03.396262+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:03.396262+0000","last_peered":"2024-09-18T12:02:03.396262+0000","last_clean":"2024-09-18T12:02:03.396262+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:03.396262+0000","last_undegraded":"2024-09-18T12:02:03.396262+0000","last_fullsized":"2024-09-18T12:02:03.396262+0000","mapping_epoch":29,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16474112,"num_objects":4022,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12602,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15492","reported_seq":15682,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.613901+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:03.613901+0000","last_peered":"2024-09-18T12:02:03.613901+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:03.613901+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16650264,"num_objects":4066,"num_object_clones":0,"num_object_copies":8132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12350,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":530}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15330","reported_seq":15363,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:03.466781+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:03.466781+0000","last_peered":"2024-09-18T12:02:03.466781+0000","last_clean":"2024-09-18T12:02:03.466781+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:03.466781+0000","last_undegraded":"2024-09-18T12:02:03.466781+0000","last_fullsized":"2024-09-18T12:02:03.466781+0000","mapping_epoch":29,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16097280,"num_objects":3930,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15330,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15406","reported_seq":15601,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:03.557131+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:03.557131+0000","last_peered":"2024-09-18T12:02:03.557131+0000","last_clean":"2024-09-18T12:02:03.557131+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:03.557131+0000","last_undegraded":"2024-09-18T12:02:03.557131+0000","last_fullsized":"2024-09-18T12:02:03.557131+0000","mapping_epoch":29,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16551936,"num_objects":4041,"num_object_clones":0,"num_object_copies":8082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15371","reported_seq":15455,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.563690+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:03.563690+0000","last_peered":"2024-09-18T12:02:03.563690+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:03.563690+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16598226,"num_objects":4054,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15369,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15405","reported_seq":15496,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.577463+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:03.577463+0000","last_peered":"2024-09-18T12:02:03.577463+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:03.577463+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16250066,"num_objects":3969,"num_object_clones":0,"num_object_copies":7938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15403,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15566","reported_seq":15795,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:03.484367+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:03.484367+0000","last_peered":"2024-09-18T12:02:03.484367+0000","last_clean":"2024-09-18T12:02:03.484367+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:03.484367+0000","last_undegraded":"2024-09-18T12:02:03.484367+0000","last_fullsized":"2024-09-18T12:02:03.484367+0000","mapping_epoch":29,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16531456,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11883,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15394","reported_seq":22526,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:04.872414+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:04.872414+0000","last_peered":"2024-09-18T12:02:04.872414+0000","last_clean":"2024-09-18T12:02:04.872414+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:04.872414+0000","last_undegraded":"2024-09-18T12:02:04.872414+0000","last_fullsized":"2024-09-18T12:02:04.872414+0000","mapping_epoch":38,"log_start":"37'5300","ondisk_log_start":"37'5300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16430290,"num_objects":4013,"num_object_clones":0,"num_object_copies":8026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15392,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15471","reported_seq":16163,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.632262+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:03.632262+0000","last_peered":"2024-09-18T12:02:03.632262+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:03.632262+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16282834,"num_objects":3977,"num_object_clones":0,"num_object_copies":7954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15469,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15474","reported_seq":15534,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.588268+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:03.588268+0000","last_peered":"2024-09-18T12:02:03.588268+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:03.588268+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16340178,"num_objects":3991,"num_object_clones":0,"num_object_copies":7982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15472,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":441}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15479","reported_seq":22892,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:03.608055+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:03.608055+0000","last_peered":"2024-09-18T12:02:03.608055+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:03.608055+0000","last_undegraded":"2024-09-18T12:02:03.608055+0000","last_fullsized":"2024-09-18T12:02:03.608055+0000","mapping_epoch":42,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16461824,"num_objects":4019,"num_object_clones":0,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11959,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":4019}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15424","reported_seq":15509,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.586384+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:03.586384+0000","last_peered":"2024-09-18T12:02:03.586384+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:03.586384+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16810601,"num_objects":4105,"num_object_clones":0,"num_object_copies":8210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15423,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15579","reported_seq":22729,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:04.836043+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:04.836043+0000","last_peered":"2024-09-18T12:02:04.836043+0000","last_clean":"2024-09-18T12:02:04.836043+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:04.836043+0000","last_undegraded":"2024-09-18T12:02:04.836043+0000","last_fullsized":"2024-09-18T12:02:04.836043+0000","mapping_epoch":40,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16152379,"num_objects":3946,"num_object_clones":0,"num_object_copies":7892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15576,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15295","reported_seq":15380,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.508388+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:03.508388+0000","last_peered":"2024-09-18T12:02:03.508388+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:03.508388+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5200","ondisk_log_start":"37'5200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15987922,"num_objects":3905,"num_object_clones":0,"num_object_copies":7810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15293,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15519","reported_seq":15608,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:03.585264+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:03.585264+0000","last_peered":"2024-09-18T12:02:03.585264+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:03.585264+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16357796,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15516,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":474}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":262287436,"num_objects":64053,"num_object_clones":0,"num_object_copies":128106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231162,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10929,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":697663488,"data_stored":697599680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160869,"ondisk_log_size":160869,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051682,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544700,"kb_used_data":242676,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93827140,"statfs":{"total":96636764160,"available":96078991360,"internally_reserved":0,"allocated":248500224,"data_stored":248251589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.626},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.438},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.66300000000000003}]}]},{"osd":1,"up_from":35,"seq":150323855445,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493564,"kb_used_data":191536,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93878276,"statfs":{"total":96636764160,"available":96131354624,"internally_reserved":0,"allocated":196132864,"data_stored":195835380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,2,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.311},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.34899999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559252,"kb_used_data":248964,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93812588,"statfs":{"total":96636764160,"available":96064090112,"internally_reserved":0,"allocated":254939136,"data_stored":254690951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,2,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":3.508},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.27300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":10.525},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":254443520,"data_stored":254440041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195211264,"data_stored":195158486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":248008704,"data_stored":248001153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:12.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:12.794 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:12.801 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-18T12:02:11.218906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":253370463,"num_objects":61878,"num_object_clones":0,"num_object_copies":123756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233341,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10995,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1580416,"kb_used_data":666060,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281535104,"statfs":{"total":289910292480,"available":288291946496,"internally_reserved":0,"allocated":682045440,"data_stored":681251136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,6,7,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18890752,"num_objects":-4612,"num_object_clones":0,"num_object_copies":-9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4612,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002580"},"pg_stats":[{"pgid":"2.f","version":"43'15677","reported_seq":15752,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.538645+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:08.538645+0000","last_peered":"2024-09-18T12:02:08.538645+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:08.538645+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15806464,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15677,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":260}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15534","reported_seq":15696,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:08.592207+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:08.592207+0000","last_peered":"2024-09-18T12:02:08.592207+0000","last_clean":"2024-09-18T12:02:08.592207+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:08.592207+0000","last_undegraded":"2024-09-18T12:02:08.592207+0000","last_fullsized":"2024-09-18T12:02:08.592207+0000","mapping_epoch":29,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15966208,"num_objects":3898,"num_object_clones":0,"num_object_copies":7796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15639","reported_seq":15828,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.611526+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:08.611526+0000","last_peered":"2024-09-18T12:02:08.611526+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:08.611526+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16048152,"num_objects":3919,"num_object_clones":0,"num_object_copies":7838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12497,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15456","reported_seq":15488,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:08.578175+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:08.578175+0000","last_peered":"2024-09-18T12:02:08.578175+0000","last_clean":"2024-09-18T12:02:08.578175+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:08.578175+0000","last_undegraded":"2024-09-18T12:02:08.578175+0000","last_fullsized":"2024-09-18T12:02:08.578175+0000","mapping_epoch":29,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15581184,"num_objects":3804,"num_object_clones":0,"num_object_copies":7608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15456,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15547","reported_seq":15742,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:08.507854+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:08.507854+0000","last_peered":"2024-09-18T12:02:08.507854+0000","last_clean":"2024-09-18T12:02:08.507854+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:08.507854+0000","last_undegraded":"2024-09-18T12:02:08.507854+0000","last_fullsized":"2024-09-18T12:02:08.507854+0000","mapping_epoch":29,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15974400,"num_objects":3900,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12712,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15515","reported_seq":15598,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.607017+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:08.607017+0000","last_peered":"2024-09-18T12:02:08.607017+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:08.607017+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16008402,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15513,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15558","reported_seq":15649,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.548370+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:08.548370+0000","last_peered":"2024-09-18T12:02:08.548370+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:08.548370+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15623378,"num_objects":3816,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15556,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15689","reported_seq":15918,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:08.600614+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:08.600614+0000","last_peered":"2024-09-18T12:02:08.600614+0000","last_clean":"2024-09-18T12:02:08.600614+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:08.600614+0000","last_undegraded":"2024-09-18T12:02:08.600614+0000","last_fullsized":"2024-09-18T12:02:08.600614+0000","mapping_epoch":29,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16027648,"num_objects":3913,"num_object_clones":0,"num_object_copies":7826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12006,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15541","reported_seq":22673,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:09.916322+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:09.916322+0000","last_peered":"2024-09-18T12:02:09.916322+0000","last_clean":"2024-09-18T12:02:09.916322+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:09.916322+0000","last_undegraded":"2024-09-18T12:02:09.916322+0000","last_fullsized":"2024-09-18T12:02:09.916322+0000","mapping_epoch":38,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15828178,"num_objects":3866,"num_object_clones":0,"num_object_copies":7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15539,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15600","reported_seq":16423,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.632634+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:08.632634+0000","last_peered":"2024-09-18T12:02:08.632634+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:08.632634+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15754450,"num_objects":3848,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15598,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":684}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15626","reported_seq":15686,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.485863+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:08.485863+0000","last_peered":"2024-09-18T12:02:08.485863+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:08.485863+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15717586,"num_objects":3839,"num_object_clones":0,"num_object_copies":7678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15624,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":289}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15606","reported_seq":23019,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:08.612354+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:08.612354+0000","last_peered":"2024-09-18T12:02:08.612354+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:08.612354+0000","last_undegraded":"2024-09-18T12:02:08.612354+0000","last_fullsized":"2024-09-18T12:02:08.612354+0000","mapping_epoch":42,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15941632,"num_objects":3892,"num_object_clones":0,"num_object_copies":7784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15553","reported_seq":15638,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.592298+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:08.592298+0000","last_peered":"2024-09-18T12:02:08.592298+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:08.592298+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16282217,"num_objects":3976,"num_object_clones":0,"num_object_copies":7952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15552,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":441}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15722","reported_seq":22872,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:09.839227+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:09.839227+0000","last_peered":"2024-09-18T12:02:09.839227+0000","last_clean":"2024-09-18T12:02:09.839227+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:09.839227+0000","last_undegraded":"2024-09-18T12:02:09.839227+0000","last_fullsized":"2024-09-18T12:02:09.839227+0000","mapping_epoch":40,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15566651,"num_objects":3803,"num_object_clones":0,"num_object_copies":7606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15719,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15424","reported_seq":15509,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.584950+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:08.584950+0000","last_peered":"2024-09-18T12:02:08.584950+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:08.584950+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15459538,"num_objects":3776,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15422,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15659","reported_seq":15748,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.592030+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:08.592030+0000","last_peered":"2024-09-18T12:02:08.592030+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:08.592030+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15784356,"num_objects":3857,"num_object_clones":0,"num_object_copies":7714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15656,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":253370444,"num_objects":61876,"num_object_clones":0,"num_object_copies":123752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61876,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233339,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10994,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679714816,"data_stored":679651008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160746,"ondisk_log_size":160746,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051683,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544440,"kb_used_data":238256,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93827400,"statfs":{"total":96636764160,"available":96079257600,"internally_reserved":0,"allocated":243974144,"data_stored":243725509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.24399999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.311},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.373}]}]},{"osd":1,"up_from":35,"seq":150323855446,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491276,"kb_used_data":185092,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93880564,"statfs":{"total":96636764160,"available":96133697536,"internally_reserved":0,"allocated":189534208,"data_stored":189236724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.29399999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544700,"kb_used_data":242712,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93827140,"statfs":{"total":96636764160,"available":96078991360,"internally_reserved":0,"allocated":248537088,"data_stored":248288903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.32300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.42899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247947264,"data_stored":247943785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188264448,"data_stored":188211670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":243503104,"data_stored":243495553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:12.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:13.612 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:13.621 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:13.636 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-18T12:02:13.221731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":253370463,"num_objects":61878,"num_object_clones":0,"num_object_copies":123756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233341,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10995,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1580416,"kb_used_data":666060,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281535104,"statfs":{"total":289910292480,"available":288291946496,"internally_reserved":0,"allocated":682045440,"data_stored":681251136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,6,7,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17682432,"num_objects":-4317,"num_object_clones":0,"num_object_copies":-8634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4317,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004948"},"pg_stats":[{"pgid":"2.f","version":"43'15677","reported_seq":15752,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.538645+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:08.538645+0000","last_peered":"2024-09-18T12:02:08.538645+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:08.538645+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15806464,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15677,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":260}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15534","reported_seq":15696,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:08.592207+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:08.592207+0000","last_peered":"2024-09-18T12:02:08.592207+0000","last_clean":"2024-09-18T12:02:08.592207+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:08.592207+0000","last_undegraded":"2024-09-18T12:02:08.592207+0000","last_fullsized":"2024-09-18T12:02:08.592207+0000","mapping_epoch":29,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15966208,"num_objects":3898,"num_object_clones":0,"num_object_copies":7796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15639","reported_seq":15828,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.611526+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:08.611526+0000","last_peered":"2024-09-18T12:02:08.611526+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:08.611526+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16048152,"num_objects":3919,"num_object_clones":0,"num_object_copies":7838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12497,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15456","reported_seq":15488,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:08.578175+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:08.578175+0000","last_peered":"2024-09-18T12:02:08.578175+0000","last_clean":"2024-09-18T12:02:08.578175+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:08.578175+0000","last_undegraded":"2024-09-18T12:02:08.578175+0000","last_fullsized":"2024-09-18T12:02:08.578175+0000","mapping_epoch":29,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15581184,"num_objects":3804,"num_object_clones":0,"num_object_copies":7608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15456,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15547","reported_seq":15742,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:08.507854+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:08.507854+0000","last_peered":"2024-09-18T12:02:08.507854+0000","last_clean":"2024-09-18T12:02:08.507854+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:08.507854+0000","last_undegraded":"2024-09-18T12:02:08.507854+0000","last_fullsized":"2024-09-18T12:02:08.507854+0000","mapping_epoch":29,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15974400,"num_objects":3900,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12712,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15515","reported_seq":15598,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.607017+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:08.607017+0000","last_peered":"2024-09-18T12:02:08.607017+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:08.607017+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16008402,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15513,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15558","reported_seq":15649,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.548370+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:08.548370+0000","last_peered":"2024-09-18T12:02:08.548370+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:08.548370+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15623378,"num_objects":3816,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15556,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15689","reported_seq":15918,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:08.600614+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:08.600614+0000","last_peered":"2024-09-18T12:02:08.600614+0000","last_clean":"2024-09-18T12:02:08.600614+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:08.600614+0000","last_undegraded":"2024-09-18T12:02:08.600614+0000","last_fullsized":"2024-09-18T12:02:08.600614+0000","mapping_epoch":29,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16027648,"num_objects":3913,"num_object_clones":0,"num_object_copies":7826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12006,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15541","reported_seq":22673,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:09.916322+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:09.916322+0000","last_peered":"2024-09-18T12:02:09.916322+0000","last_clean":"2024-09-18T12:02:09.916322+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:09.916322+0000","last_undegraded":"2024-09-18T12:02:09.916322+0000","last_fullsized":"2024-09-18T12:02:09.916322+0000","mapping_epoch":38,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15828178,"num_objects":3866,"num_object_clones":0,"num_object_copies":7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15539,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15600","reported_seq":16423,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.632634+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:08.632634+0000","last_peered":"2024-09-18T12:02:08.632634+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:08.632634+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15754450,"num_objects":3848,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15598,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":684}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15626","reported_seq":15686,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.485863+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:08.485863+0000","last_peered":"2024-09-18T12:02:08.485863+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:08.485863+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15717586,"num_objects":3839,"num_object_clones":0,"num_object_copies":7678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15624,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":289}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15606","reported_seq":23019,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:08.612354+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:08.612354+0000","last_peered":"2024-09-18T12:02:08.612354+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:08.612354+0000","last_undegraded":"2024-09-18T12:02:08.612354+0000","last_fullsized":"2024-09-18T12:02:08.612354+0000","mapping_epoch":42,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15941632,"num_objects":3892,"num_object_clones":0,"num_object_copies":7784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15553","reported_seq":15638,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.592298+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:08.592298+0000","last_peered":"2024-09-18T12:02:08.592298+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:08.592298+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16282217,"num_objects":3976,"num_object_clones":0,"num_object_copies":7952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15552,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":441}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15722","reported_seq":22872,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:09.839227+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:09.839227+0000","last_peered":"2024-09-18T12:02:09.839227+0000","last_clean":"2024-09-18T12:02:09.839227+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:09.839227+0000","last_undegraded":"2024-09-18T12:02:09.839227+0000","last_fullsized":"2024-09-18T12:02:09.839227+0000","mapping_epoch":40,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15566651,"num_objects":3803,"num_object_clones":0,"num_object_copies":7606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15719,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15424","reported_seq":15509,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.584950+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:08.584950+0000","last_peered":"2024-09-18T12:02:08.584950+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:08.584950+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15459538,"num_objects":3776,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15422,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15659","reported_seq":15748,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.592030+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:08.592030+0000","last_peered":"2024-09-18T12:02:08.592030+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:08.592030+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15784356,"num_objects":3857,"num_object_clones":0,"num_object_copies":7714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15656,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":253370444,"num_objects":61876,"num_object_clones":0,"num_object_copies":123752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61876,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233339,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10994,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679714816,"data_stored":679651008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160746,"ondisk_log_size":160746,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051683,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544440,"kb_used_data":238256,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93827400,"statfs":{"total":96636764160,"available":96079257600,"internally_reserved":0,"allocated":243974144,"data_stored":243725509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.24399999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.311},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.373}]}]},{"osd":1,"up_from":35,"seq":150323855446,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491276,"kb_used_data":185092,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93880564,"statfs":{"total":96636764160,"available":96133697536,"internally_reserved":0,"allocated":189534208,"data_stored":189236724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.29399999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544700,"kb_used_data":242712,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93827140,"statfs":{"total":96636764160,"available":96078991360,"internally_reserved":0,"allocated":248537088,"data_stored":248288903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.32300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.42899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247947264,"data_stored":247943785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188264448,"data_stored":188211670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":243503104,"data_stored":243495553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:13.637 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:02:13.637 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'15677", 'reported_seq': 15752, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:08.538645+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:02:08.538645+0000', 'last_peered': '2024-09-18T12:02:08.538645+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:02:08.538645+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'5600", 'ondisk_log_start': "37'5600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15806464, 'num_objects': 3859, 'num_object_clones': 0, 'num_object_copies': 7718, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15677, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 260}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:13.638 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:02:13.638 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'15639", 'reported_seq': 15828, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:08.611526+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:02:08.611526+0000', 'last_peered': '2024-09-18T12:02:08.611526+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:02:08.611526+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'5600", 'ondisk_log_start': "37'5600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16048152, 'num_objects': 3919, 'num_object_clones': 0, 'num_object_copies': 7838, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3919, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12497, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 383}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:13.638 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:02:13.638 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'15515", 'reported_seq': 15598, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:08.607017+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:02:08.607017+0000', 'last_peered': '2024-09-18T12:02:08.607017+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:02:08.607017+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'5500", 'ondisk_log_start': "37'5500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16008402, 'num_objects': 3910, 'num_object_clones': 0, 'num_object_copies': 7820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15513, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 383}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:13.638 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:02:13.638 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'15558", 'reported_seq': 15649, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:08.548370+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:02:08.548370+0000', 'last_peered': '2024-09-18T12:02:08.548370+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:02:08.548370+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'5500", 'ondisk_log_start': "37'5500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15623378, 'num_objects': 3816, 'num_object_clones': 0, 'num_object_copies': 7632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3816, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15556, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 257}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:13.638 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:02:13.638 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'15600", 'reported_seq': 16423, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:08.632634+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:02:08.632634+0000', 'last_peered': '2024-09-18T12:02:08.632634+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:02:08.632634+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'5500", 'ondisk_log_start': "37'5500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15754450, 'num_objects': 3848, 'num_object_clones': 0, 'num_object_copies': 7696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15598, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 372, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 684}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:13.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:02:13.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'15626", 'reported_seq': 15686, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:08.485863+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:02:08.485863+0000', 'last_peered': '2024-09-18T12:02:08.485863+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:02:08.485863+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'5600", 'ondisk_log_start': "37'5600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15717586, 'num_objects': 3839, 'num_object_clones': 0, 'num_object_copies': 7678, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15624, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 289}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:13.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:02:13.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'15606", 'reported_seq': 23019, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:02:08.612354+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:02:08.612354+0000', 'last_peered': '2024-09-18T12:02:08.612354+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:02:08.612354+0000', 'last_undegraded': '2024-09-18T12:02:08.612354+0000', 'last_fullsized': '2024-09-18T12:02:08.612354+0000', 'mapping_epoch': 42, 'log_start': "37'5600", 'ondisk_log_start': "37'5600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15941632, 'num_objects': 3892, 'num_object_clones': 0, 'num_object_copies': 7784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12086, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 3892}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:13.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:02:13.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'15553", 'reported_seq': 15638, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:08.592298+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:02:08.592298+0000', 'last_peered': '2024-09-18T12:02:08.592298+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:02:08.592298+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'5500", 'ondisk_log_start': "37'5500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16282217, 'num_objects': 3976, 'num_object_clones': 0, 'num_object_copies': 7952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15552, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 441}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:13.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:02:13.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'15424", 'reported_seq': 15509, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:08.584950+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:02:08.584950+0000', 'last_peered': '2024-09-18T12:02:08.584950+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:02:08.584950+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'5400", 'ondisk_log_start': "37'5400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15459538, 'num_objects': 3776, 'num_object_clones': 0, 'num_object_copies': 7552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15422, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 183}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:13.640 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:02:13.640 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'15659", 'reported_seq': 15748, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:08.592030+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:02:08.592030+0000', 'last_peered': '2024-09-18T12:02:08.592030+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:02:08.592030+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'5600", 'ondisk_log_start': "37'5600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15784356, 'num_objects': 3857, 'num_object_clones': 0, 'num_object_copies': 7714, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15656, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 334}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:13.640 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:02:14.345 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:14.345 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31599/123756 objects degraded (25.533%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":534,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":61878,"data_bytes":253370463,"bytes_used":1618345984,"bytes_avail":288291946496,"bytes_total":289910292480,"degraded_objects":31599,"degraded_total":123756,"degraded_ratio":0.25533307476001166,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":359},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:02:14.363 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:02:14.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:15.156 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:15.187 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:15.217 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-18T12:02:13.221731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":253370463,"num_objects":61878,"num_object_clones":0,"num_object_copies":123756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233341,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10995,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1580416,"kb_used_data":666060,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281535104,"statfs":{"total":289910292480,"available":288291946496,"internally_reserved":0,"allocated":682045440,"data_stored":681251136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,6,7,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17682432,"num_objects":-4317,"num_object_clones":0,"num_object_copies":-8634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4317,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004948"},"pg_stats":[{"pgid":"2.f","version":"43'15677","reported_seq":15752,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.538645+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:08.538645+0000","last_peered":"2024-09-18T12:02:08.538645+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:08.538645+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15806464,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15677,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":260}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15534","reported_seq":15696,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:08.592207+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:08.592207+0000","last_peered":"2024-09-18T12:02:08.592207+0000","last_clean":"2024-09-18T12:02:08.592207+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:08.592207+0000","last_undegraded":"2024-09-18T12:02:08.592207+0000","last_fullsized":"2024-09-18T12:02:08.592207+0000","mapping_epoch":29,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15966208,"num_objects":3898,"num_object_clones":0,"num_object_copies":7796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15639","reported_seq":15828,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.611526+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:08.611526+0000","last_peered":"2024-09-18T12:02:08.611526+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:08.611526+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16048152,"num_objects":3919,"num_object_clones":0,"num_object_copies":7838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12497,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15456","reported_seq":15488,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:08.578175+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:08.578175+0000","last_peered":"2024-09-18T12:02:08.578175+0000","last_clean":"2024-09-18T12:02:08.578175+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:08.578175+0000","last_undegraded":"2024-09-18T12:02:08.578175+0000","last_fullsized":"2024-09-18T12:02:08.578175+0000","mapping_epoch":29,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15581184,"num_objects":3804,"num_object_clones":0,"num_object_copies":7608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15456,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15547","reported_seq":15742,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:08.507854+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:08.507854+0000","last_peered":"2024-09-18T12:02:08.507854+0000","last_clean":"2024-09-18T12:02:08.507854+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:08.507854+0000","last_undegraded":"2024-09-18T12:02:08.507854+0000","last_fullsized":"2024-09-18T12:02:08.507854+0000","mapping_epoch":29,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15974400,"num_objects":3900,"num_object_clones":0,"num_object_copies":7800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12712,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15515","reported_seq":15598,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.607017+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:08.607017+0000","last_peered":"2024-09-18T12:02:08.607017+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:08.607017+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16008402,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15513,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":383}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15558","reported_seq":15649,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.548370+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:08.548370+0000","last_peered":"2024-09-18T12:02:08.548370+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:08.548370+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15623378,"num_objects":3816,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15556,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15689","reported_seq":15918,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:08.600614+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:08.600614+0000","last_peered":"2024-09-18T12:02:08.600614+0000","last_clean":"2024-09-18T12:02:08.600614+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:08.600614+0000","last_undegraded":"2024-09-18T12:02:08.600614+0000","last_fullsized":"2024-09-18T12:02:08.600614+0000","mapping_epoch":29,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16027648,"num_objects":3913,"num_object_clones":0,"num_object_copies":7826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12006,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15541","reported_seq":22673,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:09.916322+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:09.916322+0000","last_peered":"2024-09-18T12:02:09.916322+0000","last_clean":"2024-09-18T12:02:09.916322+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:09.916322+0000","last_undegraded":"2024-09-18T12:02:09.916322+0000","last_fullsized":"2024-09-18T12:02:09.916322+0000","mapping_epoch":38,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15828178,"num_objects":3866,"num_object_clones":0,"num_object_copies":7732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15539,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15600","reported_seq":16423,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.632634+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:08.632634+0000","last_peered":"2024-09-18T12:02:08.632634+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:08.632634+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15754450,"num_objects":3848,"num_object_clones":0,"num_object_copies":7696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15598,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":684}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15626","reported_seq":15686,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.485863+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:08.485863+0000","last_peered":"2024-09-18T12:02:08.485863+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:08.485863+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15717586,"num_objects":3839,"num_object_clones":0,"num_object_copies":7678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15624,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":289}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15606","reported_seq":23019,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:08.612354+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:08.612354+0000","last_peered":"2024-09-18T12:02:08.612354+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:08.612354+0000","last_undegraded":"2024-09-18T12:02:08.612354+0000","last_fullsized":"2024-09-18T12:02:08.612354+0000","mapping_epoch":42,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15941632,"num_objects":3892,"num_object_clones":0,"num_object_copies":7784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12086,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15553","reported_seq":15638,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.592298+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:08.592298+0000","last_peered":"2024-09-18T12:02:08.592298+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:08.592298+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16282217,"num_objects":3976,"num_object_clones":0,"num_object_copies":7952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15552,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":441}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15722","reported_seq":22872,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:09.839227+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:09.839227+0000","last_peered":"2024-09-18T12:02:09.839227+0000","last_clean":"2024-09-18T12:02:09.839227+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:09.839227+0000","last_undegraded":"2024-09-18T12:02:09.839227+0000","last_fullsized":"2024-09-18T12:02:09.839227+0000","mapping_epoch":40,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15566651,"num_objects":3803,"num_object_clones":0,"num_object_copies":7606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15719,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15424","reported_seq":15509,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.584950+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:08.584950+0000","last_peered":"2024-09-18T12:02:08.584950+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:08.584950+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5400","ondisk_log_start":"37'5400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15459538,"num_objects":3776,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15422,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15659","reported_seq":15748,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:08.592030+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:08.592030+0000","last_peered":"2024-09-18T12:02:08.592030+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:08.592030+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15784356,"num_objects":3857,"num_object_clones":0,"num_object_copies":7714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15656,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":253370444,"num_objects":61876,"num_object_clones":0,"num_object_copies":123752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61876,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233339,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10994,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679714816,"data_stored":679651008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160746,"ondisk_log_size":160746,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051683,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544440,"kb_used_data":238256,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93827400,"statfs":{"total":96636764160,"available":96079257600,"internally_reserved":0,"allocated":243974144,"data_stored":243725509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.24399999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.311},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.373}]}]},{"osd":1,"up_from":35,"seq":150323855446,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491276,"kb_used_data":185092,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93880564,"statfs":{"total":96636764160,"available":96133697536,"internally_reserved":0,"allocated":189534208,"data_stored":189236724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.29399999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544700,"kb_used_data":242712,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93827140,"statfs":{"total":96636764160,"available":96078991360,"internally_reserved":0,"allocated":248537088,"data_stored":248288903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.32300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.42899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":247947264,"data_stored":247943785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188264448,"data_stored":188211670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":243503104,"data_stored":243495553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:18.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:18.885 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:18.890 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:18.903 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-18T12:02:17.223513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":244408415,"num_objects":59690,"num_object_clones":0,"num_object_copies":119380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":235529,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11064,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161037,"ondisk_log_size":161037,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1579640,"kb_used_data":648640,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":281535880,"statfs":{"total":289910292480,"available":288292741120,"internally_reserved":0,"allocated":664207360,"data_stored":663413056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,6,6,7,7,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17879040,"num_objects":-4365,"num_object_clones":0,"num_object_copies":-8730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005897"},"pg_stats":[{"pgid":"2.f","version":"43'15829","reported_seq":15902,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:13.612195+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:13.612195+0000","last_peered":"2024-09-18T12:02:13.612195+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:13.612195+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15183872,"num_objects":3707,"num_object_clones":0,"num_object_copies":7414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15829,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":108}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15673","reported_seq":15835,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:13.595382+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:13.595382+0000","last_peered":"2024-09-18T12:02:13.595382+0000","last_clean":"2024-09-18T12:02:13.595382+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:13.595382+0000","last_undegraded":"2024-09-18T12:02:13.595382+0000","last_fullsized":"2024-09-18T12:02:13.595382+0000","mapping_epoch":29,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15396864,"num_objects":3759,"num_object_clones":0,"num_object_copies":7518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12865,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15770","reported_seq":15959,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:13.648711+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:13.648711+0000","last_peered":"2024-09-18T12:02:13.648711+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:13.648711+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15511576,"num_objects":3788,"num_object_clones":0,"num_object_copies":7576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3788,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12628,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15581","reported_seq":15614,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:13.572277+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:13.572277+0000","last_peered":"2024-09-18T12:02:13.572277+0000","last_clean":"2024-09-18T12:02:13.572277+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:13.572277+0000","last_undegraded":"2024-09-18T12:02:13.572277+0000","last_fullsized":"2024-09-18T12:02:13.572277+0000","mapping_epoch":29,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15069184,"num_objects":3679,"num_object_clones":0,"num_object_copies":7358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15581,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15679","reported_seq":15874,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:13.485491+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:13.485491+0000","last_peered":"2024-09-18T12:02:13.485491+0000","last_clean":"2024-09-18T12:02:13.485491+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:13.485491+0000","last_undegraded":"2024-09-18T12:02:13.485491+0000","last_fullsized":"2024-09-18T12:02:13.485491+0000","mapping_epoch":29,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15433728,"num_objects":3768,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12844,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15650","reported_seq":15735,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:13.655616+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:13.655616+0000","last_peered":"2024-09-18T12:02:13.655616+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:13.655616+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15455442,"num_objects":3775,"num_object_clones":0,"num_object_copies":7550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15648,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":248}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15683","reported_seq":15774,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:13.559709+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:13.559709+0000","last_peered":"2024-09-18T12:02:13.559709+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:13.559709+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15111378,"num_objects":3691,"num_object_clones":0,"num_object_copies":7382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15681,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15807","reported_seq":16035,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:13.612396+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:13.612396+0000","last_peered":"2024-09-18T12:02:13.612396+0000","last_clean":"2024-09-18T12:02:13.612396+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:13.612396+0000","last_undegraded":"2024-09-18T12:02:13.612396+0000","last_fullsized":"2024-09-18T12:02:13.612396+0000","mapping_epoch":29,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15544320,"num_objects":3795,"num_object_clones":0,"num_object_copies":7590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12124,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15671","reported_seq":22802,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:14.963911+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:14.963911+0000","last_peered":"2024-09-18T12:02:14.963911+0000","last_clean":"2024-09-18T12:02:14.963911+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:14.963911+0000","last_undegraded":"2024-09-18T12:02:14.963911+0000","last_fullsized":"2024-09-18T12:02:14.963911+0000","mapping_epoch":38,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15295698,"num_objects":3736,"num_object_clones":0,"num_object_copies":7472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15669,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15755","reported_seq":16716,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:13.656558+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:13.656558+0000","last_peered":"2024-09-18T12:02:13.656558+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:13.656558+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15119570,"num_objects":3693,"num_object_clones":0,"num_object_copies":7386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15753,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":598}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15764","reported_seq":15824,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:13.558959+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:13.558959+0000","last_peered":"2024-09-18T12:02:13.558959+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:13.558959+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15152338,"num_objects":3701,"num_object_clones":0,"num_object_copies":7402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15762,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":151}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15741","reported_seq":23154,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:13.612536+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:13.612536+0000","last_peered":"2024-09-18T12:02:13.612536+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:13.612536+0000","last_undegraded":"2024-09-18T12:02:13.612536+0000","last_fullsized":"2024-09-18T12:02:13.612536+0000","mapping_epoch":42,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15388672,"num_objects":3757,"num_object_clones":0,"num_object_copies":7514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12221,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3757}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15693","reported_seq":15775,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:13.638017+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:13.638017+0000","last_peered":"2024-09-18T12:02:13.638017+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:13.638017+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15708777,"num_objects":3836,"num_object_clones":0,"num_object_copies":7672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15692,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15861","reported_seq":23011,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:14.992869+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:14.992869+0000","last_peered":"2024-09-18T12:02:14.992869+0000","last_clean":"2024-09-18T12:02:14.992869+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:14.992869+0000","last_undegraded":"2024-09-18T12:02:14.992869+0000","last_fullsized":"2024-09-18T12:02:14.992869+0000","mapping_epoch":40,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14997307,"num_objects":3664,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15858,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15585","reported_seq":15670,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:13.570246+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:13.570246+0000","last_peered":"2024-09-18T12:02:13.570246+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:13.570246+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5500","ondisk_log_start":"37'5500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14800082,"num_objects":3615,"num_object_clones":0,"num_object_copies":7230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15583,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":22}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15792","reported_seq":15881,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:13.595894+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:13.595894+0000","last_peered":"2024-09-18T12:02:13.595894+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:13.595894+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15239588,"num_objects":3724,"num_object_clones":0,"num_object_copies":7448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15789,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":244408396,"num_objects":59688,"num_object_clones":0,"num_object_copies":119376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59688,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":235527,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11063,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":661921792,"data_stored":661857984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161034,"ondisk_log_size":161034,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051684,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544472,"kb_used_data":234128,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93827368,"statfs":{"total":96636764160,"available":96079224832,"internally_reserved":0,"allocated":239747072,"data_stored":239498437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.245},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":2.7759999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5720000000000001,"15min":1.9119999999999999},"min":{"1min":0.23100000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.7279999999999998,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":2.8199999999999998},{"interface":"front","average":{"1min":1.012,"5min":1.173,"15min":0.96999999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.2539999999999996,"5min":11.109,"15min":11.109},"last":0.28599999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855447,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492948,"kb_used_data":178420,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93878892,"statfs":{"total":96636764160,"available":96131985408,"internally_reserved":0,"allocated":182702080,"data_stored":182404596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,3,5,1,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.29199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542220,"kb_used_data":236092,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93829620,"statfs":{"total":96636764160,"available":96081530880,"internally_reserved":0,"allocated":241758208,"data_stored":241510023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,1,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.317}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.22900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241160192,"data_stored":241156713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":181579776,"data_stored":181526998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":239181824,"data_stored":239174273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:18.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:19.613 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:19.622 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:19.662 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-09-18T12:02:19.223943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":236449887,"num_objects":57747,"num_object_clones":0,"num_object_copies":115494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57747,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237472,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11134,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160780,"ondisk_log_size":160780,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1560164,"kb_used_data":637480,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281555356,"statfs":{"total":289910292480,"available":288312684544,"internally_reserved":0,"allocated":652779520,"data_stored":651985216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,5,9,12,4,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25837568,"num_objects":-6308,"num_object_clones":0,"num_object_copies":-12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6308,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005913"},"pg_stats":[{"pgid":"2.f","version":"43'15966","reported_seq":16041,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.599958+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:18.599958+0000","last_peered":"2024-09-18T12:02:18.599958+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:18.599958+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14622720,"num_objects":3570,"num_object_clones":0,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15966,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-29}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15820","reported_seq":15981,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:18.629883+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:18.629883+0000","last_peered":"2024-09-18T12:02:18.629883+0000","last_clean":"2024-09-18T12:02:18.629883+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:18.629883+0000","last_undegraded":"2024-09-18T12:02:18.629883+0000","last_fullsized":"2024-09-18T12:02:18.629883+0000","mapping_epoch":29,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14794752,"num_objects":3612,"num_object_clones":0,"num_object_copies":7224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15894","reported_seq":16084,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.684972+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:18.684972+0000","last_peered":"2024-09-18T12:02:18.684972+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:18.684972+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15003672,"num_objects":3664,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12752,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15708","reported_seq":15741,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:18.600549+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:18.600549+0000","last_peered":"2024-09-18T12:02:18.600549+0000","last_clean":"2024-09-18T12:02:18.600549+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:18.600549+0000","last_undegraded":"2024-09-18T12:02:18.600549+0000","last_fullsized":"2024-09-18T12:02:18.600549+0000","mapping_epoch":29,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14548992,"num_objects":3552,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15708,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15822","reported_seq":16017,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:18.625649+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:18.625649+0000","last_peered":"2024-09-18T12:02:18.625649+0000","last_clean":"2024-09-18T12:02:18.625649+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:18.625649+0000","last_undegraded":"2024-09-18T12:02:18.625649+0000","last_fullsized":"2024-09-18T12:02:18.625649+0000","mapping_epoch":29,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14848000,"num_objects":3625,"num_object_clones":0,"num_object_copies":7250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12987,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15786","reported_seq":15868,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.715601+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:18.715601+0000","last_peered":"2024-09-18T12:02:18.715601+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:18.715601+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14898386,"num_objects":3639,"num_object_clones":0,"num_object_copies":7278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15784,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":112}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15837","reported_seq":15928,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.668123+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:18.668123+0000","last_peered":"2024-09-18T12:02:18.668123+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:18.668123+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14480594,"num_objects":3537,"num_object_clones":0,"num_object_copies":7074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15835,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-22}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15960","reported_seq":16188,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:18.695624+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:18.695624+0000","last_peered":"2024-09-18T12:02:18.695624+0000","last_clean":"2024-09-18T12:02:18.695624+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:18.695624+0000","last_undegraded":"2024-09-18T12:02:18.695624+0000","last_fullsized":"2024-09-18T12:02:18.695624+0000","mapping_epoch":29,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14917632,"num_objects":3642,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12277,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15671","reported_seq":22802,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:14.963911+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:14.963911+0000","last_peered":"2024-09-18T12:02:14.963911+0000","last_clean":"2024-09-18T12:02:14.963911+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:14.963911+0000","last_undegraded":"2024-09-18T12:02:14.963911+0000","last_fullsized":"2024-09-18T12:02:14.963911+0000","mapping_epoch":38,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15295698,"num_objects":3736,"num_object_clones":0,"num_object_copies":7472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15669,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15889","reported_seq":16988,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.651644+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:18.651644+0000","last_peered":"2024-09-18T12:02:18.651644+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:18.651644+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14570706,"num_objects":3559,"num_object_clones":0,"num_object_copies":7118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15887,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":534}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15906","reported_seq":15966,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.640881+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:18.640881+0000","last_peered":"2024-09-18T12:02:18.640881+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:18.640881+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14570706,"num_objects":3559,"num_object_clones":0,"num_object_copies":7118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15904,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":9}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15883","reported_seq":23296,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:18.511522+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:18.511522+0000","last_peered":"2024-09-18T12:02:18.511522+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:18.511522+0000","last_undegraded":"2024-09-18T12:02:18.511522+0000","last_fullsized":"2024-09-18T12:02:18.511522+0000","mapping_epoch":42,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14807040,"num_objects":3615,"num_object_clones":0,"num_object_copies":7230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3615}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15831","reported_seq":15917,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.632902+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:18.632902+0000","last_peered":"2024-09-18T12:02:18.632902+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:18.632902+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15143529,"num_objects":3698,"num_object_clones":0,"num_object_copies":7396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15830,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":163}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15861","reported_seq":23011,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:14.992869+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:14.992869+0000","last_peered":"2024-09-18T12:02:14.992869+0000","last_clean":"2024-09-18T12:02:14.992869+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:14.992869+0000","last_undegraded":"2024-09-18T12:02:14.992869+0000","last_fullsized":"2024-09-18T12:02:14.992869+0000","mapping_epoch":40,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14997307,"num_objects":3664,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15858,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15715","reported_seq":15800,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.639544+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:18.639544+0000","last_peered":"2024-09-18T12:02:18.639544+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:18.639544+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14267602,"num_objects":3485,"num_object_clones":0,"num_object_copies":6970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15713,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15928","reported_seq":16017,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.647382+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:18.647382+0000","last_peered":"2024-09-18T12:02:18.647382+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:18.647382+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14682532,"num_objects":3588,"num_object_clones":0,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15925,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":236449868,"num_objects":57745,"num_object_clones":0,"num_object_copies":115490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237470,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11133,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":650506240,"data_stored":650442432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051685,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527684,"kb_used_data":229816,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93844156,"statfs":{"total":96636764160,"available":96096415744,"internally_reserved":0,"allocated":235331584,"data_stored":235082949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.432}]}]},{"osd":1,"up_from":35,"seq":150323855447,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492948,"kb_used_data":178420,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93878892,"statfs":{"total":96636764160,"available":96131985408,"internally_reserved":0,"allocated":182702080,"data_stored":182404596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,3,5,1,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.29199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539532,"kb_used_data":229244,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93832308,"statfs":{"total":96636764160,"available":96084283392,"internally_reserved":0,"allocated":234745856,"data_stored":234497671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.503},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.33500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.53600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234336256,"data_stored":234332777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":181579776,"data_stored":181526998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":234590208,"data_stored":234582657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:19.664 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:02:19.664 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'15966", 'reported_seq': 16041, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:18.599958+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:02:18.599958+0000', 'last_peered': '2024-09-18T12:02:18.599958+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:02:18.599958+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'5900", 'ondisk_log_start': "37'5900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14622720, 'num_objects': 3570, 'num_object_clones': 0, 'num_object_copies': 7140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15966, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -29}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:19.665 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:02:19.665 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'15894", 'reported_seq': 16084, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:18.684972+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:02:18.684972+0000', 'last_peered': '2024-09-18T12:02:18.684972+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:02:18.684972+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'5800", 'ondisk_log_start': "37'5800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15003672, 'num_objects': 3664, 'num_object_clones': 0, 'num_object_copies': 7328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3664, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12752, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:19.665 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:02:19.665 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'15786", 'reported_seq': 15868, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:18.715601+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:02:18.715601+0000', 'last_peered': '2024-09-18T12:02:18.715601+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:02:18.715601+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'5700", 'ondisk_log_start': "37'5700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14898386, 'num_objects': 3639, 'num_object_clones': 0, 'num_object_copies': 7278, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3639, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15784, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 112}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:19.665 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:02:19.665 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'15837", 'reported_seq': 15928, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:18.668123+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:02:18.668123+0000', 'last_peered': '2024-09-18T12:02:18.668123+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:02:18.668123+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'5800", 'ondisk_log_start': "37'5800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14480594, 'num_objects': 3537, 'num_object_clones': 0, 'num_object_copies': 7074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15835, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -22}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:19.665 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:02:19.665 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'15889", 'reported_seq': 16988, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:18.651644+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:02:18.651644+0000', 'last_peered': '2024-09-18T12:02:18.651644+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:02:18.651644+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'5800", 'ondisk_log_start': "37'5800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14570706, 'num_objects': 3559, 'num_object_clones': 0, 'num_object_copies': 7118, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3025, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15887, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 511, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 534}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:19.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:02:19.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'15906", 'reported_seq': 15966, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:18.640881+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:02:18.640881+0000', 'last_peered': '2024-09-18T12:02:18.640881+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:02:18.640881+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'5900", 'ondisk_log_start': "37'5900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14570706, 'num_objects': 3559, 'num_object_clones': 0, 'num_object_copies': 7118, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15904, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 9}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:19.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:02:19.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'15883", 'reported_seq': 23296, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:02:18.511522+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:02:18.511522+0000', 'last_peered': '2024-09-18T12:02:18.511522+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:02:18.511522+0000', 'last_undegraded': '2024-09-18T12:02:18.511522+0000', 'last_fullsized': '2024-09-18T12:02:18.511522+0000', 'mapping_epoch': 42, 'log_start': "37'5800", 'ondisk_log_start': "37'5800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14807040, 'num_objects': 3615, 'num_object_clones': 0, 'num_object_copies': 7230, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3615, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12363, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 3615}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:19.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:02:19.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'15831", 'reported_seq': 15917, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:18.632902+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:02:18.632902+0000', 'last_peered': '2024-09-18T12:02:18.632902+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:02:18.632902+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'5800", 'ondisk_log_start': "37'5800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15143529, 'num_objects': 3698, 'num_object_clones': 0, 'num_object_copies': 7396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15830, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 163}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:19.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:02:19.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'15715", 'reported_seq': 15800, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:18.639544+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:02:18.639544+0000', 'last_peered': '2024-09-18T12:02:18.639544+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:02:18.639544+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'5700", 'ondisk_log_start': "37'5700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14267602, 'num_objects': 3485, 'num_object_clones': 0, 'num_object_copies': 6970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15713, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:19.666 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:02:19.667 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'15928", 'reported_seq': 16017, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:18.647382+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:02:18.647382+0000', 'last_peered': '2024-09-18T12:02:18.647382+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:02:18.647382+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'5900", 'ondisk_log_start': "37'5900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14682532, 'num_objects': 3588, 'num_object_clones': 0, 'num_object_copies': 7176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15925, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 65}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:19.667 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:02:20.514 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:20.519 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31460/115494 objects degraded (27.240%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":540,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":57747,"data_bytes":236449887,"bytes_used":1597607936,"bytes_avail":288312684544,"bytes_total":289910292480,"degraded_objects":31460,"degraded_total":115494,"degraded_ratio":0.27239510277590179,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":204,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":525},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:02:20.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:02:20.539 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:21.209 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:21.238 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:21.251 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-09-18T12:02:19.223943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":236449887,"num_objects":57747,"num_object_clones":0,"num_object_copies":115494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57747,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237472,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11134,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160780,"ondisk_log_size":160780,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1560164,"kb_used_data":637480,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281555356,"statfs":{"total":289910292480,"available":288312684544,"internally_reserved":0,"allocated":652779520,"data_stored":651985216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,5,9,12,4,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25837568,"num_objects":-6308,"num_object_clones":0,"num_object_copies":-12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6308,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005913"},"pg_stats":[{"pgid":"2.f","version":"43'15966","reported_seq":16041,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.599958+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:18.599958+0000","last_peered":"2024-09-18T12:02:18.599958+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:18.599958+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14622720,"num_objects":3570,"num_object_clones":0,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15966,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-29}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15820","reported_seq":15981,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:18.629883+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:18.629883+0000","last_peered":"2024-09-18T12:02:18.629883+0000","last_clean":"2024-09-18T12:02:18.629883+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:18.629883+0000","last_undegraded":"2024-09-18T12:02:18.629883+0000","last_fullsized":"2024-09-18T12:02:18.629883+0000","mapping_epoch":29,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14794752,"num_objects":3612,"num_object_clones":0,"num_object_copies":7224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15894","reported_seq":16084,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.684972+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:18.684972+0000","last_peered":"2024-09-18T12:02:18.684972+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:18.684972+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15003672,"num_objects":3664,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12752,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15708","reported_seq":15741,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:18.600549+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:18.600549+0000","last_peered":"2024-09-18T12:02:18.600549+0000","last_clean":"2024-09-18T12:02:18.600549+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:18.600549+0000","last_undegraded":"2024-09-18T12:02:18.600549+0000","last_fullsized":"2024-09-18T12:02:18.600549+0000","mapping_epoch":29,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14548992,"num_objects":3552,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15708,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15822","reported_seq":16017,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:18.625649+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:18.625649+0000","last_peered":"2024-09-18T12:02:18.625649+0000","last_clean":"2024-09-18T12:02:18.625649+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:18.625649+0000","last_undegraded":"2024-09-18T12:02:18.625649+0000","last_fullsized":"2024-09-18T12:02:18.625649+0000","mapping_epoch":29,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14848000,"num_objects":3625,"num_object_clones":0,"num_object_copies":7250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12987,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15786","reported_seq":15868,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.715601+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:18.715601+0000","last_peered":"2024-09-18T12:02:18.715601+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:18.715601+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14898386,"num_objects":3639,"num_object_clones":0,"num_object_copies":7278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15784,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":112}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15837","reported_seq":15928,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.668123+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:18.668123+0000","last_peered":"2024-09-18T12:02:18.668123+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:18.668123+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14480594,"num_objects":3537,"num_object_clones":0,"num_object_copies":7074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15835,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-22}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15960","reported_seq":16188,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:18.695624+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:18.695624+0000","last_peered":"2024-09-18T12:02:18.695624+0000","last_clean":"2024-09-18T12:02:18.695624+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:18.695624+0000","last_undegraded":"2024-09-18T12:02:18.695624+0000","last_fullsized":"2024-09-18T12:02:18.695624+0000","mapping_epoch":29,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14917632,"num_objects":3642,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12277,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15671","reported_seq":22802,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:14.963911+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:14.963911+0000","last_peered":"2024-09-18T12:02:14.963911+0000","last_clean":"2024-09-18T12:02:14.963911+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:14.963911+0000","last_undegraded":"2024-09-18T12:02:14.963911+0000","last_fullsized":"2024-09-18T12:02:14.963911+0000","mapping_epoch":38,"log_start":"37'5600","ondisk_log_start":"37'5600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15295698,"num_objects":3736,"num_object_clones":0,"num_object_copies":7472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15669,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15889","reported_seq":16988,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.651644+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:18.651644+0000","last_peered":"2024-09-18T12:02:18.651644+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:18.651644+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14570706,"num_objects":3559,"num_object_clones":0,"num_object_copies":7118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15887,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":534}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15906","reported_seq":15966,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.640881+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:18.640881+0000","last_peered":"2024-09-18T12:02:18.640881+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:18.640881+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14570706,"num_objects":3559,"num_object_clones":0,"num_object_copies":7118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15904,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":9}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15883","reported_seq":23296,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:18.511522+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:18.511522+0000","last_peered":"2024-09-18T12:02:18.511522+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:18.511522+0000","last_undegraded":"2024-09-18T12:02:18.511522+0000","last_fullsized":"2024-09-18T12:02:18.511522+0000","mapping_epoch":42,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14807040,"num_objects":3615,"num_object_clones":0,"num_object_copies":7230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3615}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15831","reported_seq":15917,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.632902+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:18.632902+0000","last_peered":"2024-09-18T12:02:18.632902+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:18.632902+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15143529,"num_objects":3698,"num_object_clones":0,"num_object_copies":7396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15830,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":163}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15861","reported_seq":23011,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:14.992869+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:14.992869+0000","last_peered":"2024-09-18T12:02:14.992869+0000","last_clean":"2024-09-18T12:02:14.992869+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:14.992869+0000","last_undegraded":"2024-09-18T12:02:14.992869+0000","last_fullsized":"2024-09-18T12:02:14.992869+0000","mapping_epoch":40,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14997307,"num_objects":3664,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15858,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15715","reported_seq":15800,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.639544+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:18.639544+0000","last_peered":"2024-09-18T12:02:18.639544+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:18.639544+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14267602,"num_objects":3485,"num_object_clones":0,"num_object_copies":6970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15713,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15928","reported_seq":16017,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.647382+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:18.647382+0000","last_peered":"2024-09-18T12:02:18.647382+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:18.647382+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14682532,"num_objects":3588,"num_object_clones":0,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15925,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":236449868,"num_objects":57745,"num_object_clones":0,"num_object_copies":115490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237470,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11133,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":650506240,"data_stored":650442432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051685,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527684,"kb_used_data":229816,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93844156,"statfs":{"total":96636764160,"available":96096415744,"internally_reserved":0,"allocated":235331584,"data_stored":235082949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.432}]}]},{"osd":1,"up_from":35,"seq":150323855447,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492948,"kb_used_data":178420,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93878892,"statfs":{"total":96636764160,"available":96131985408,"internally_reserved":0,"allocated":182702080,"data_stored":182404596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,3,5,1,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.38500000000000001,"15min":0.443},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.6859999999999999,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6459999999999999,"15min":1.202},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":9.6940000000000008,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.25}]},{"osd":2,"last update":"Wed Sep 18 12:01:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.778,"15min":1.1910000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.0020000000000007,"5min":11.147,"15min":11.147},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.86299999999999999,"15min":0.70499999999999996},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.9029999999999996,"5min":11.996,"15min":11.996},"last":0.29199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539532,"kb_used_data":229244,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93832308,"statfs":{"total":96636764160,"available":96084283392,"internally_reserved":0,"allocated":234745856,"data_stored":234497671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.503},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.33500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.53600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234336256,"data_stored":234332777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":181579776,"data_stored":181526998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":234590208,"data_stored":234582657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:24.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:24.964 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:24.982 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:24.993 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-18T12:02:23.227278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":235331679,"num_objects":57474,"num_object_clones":0,"num_object_copies":114948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237745,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11134,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1541112,"kb_used_data":630928,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281574408,"statfs":{"total":289910292480,"available":288332193792,"internally_reserved":0,"allocated":646070272,"data_stored":645275968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,7,8,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18038784,"num_objects":-4404,"num_object_clones":0,"num_object_copies":-8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4404,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008371"},"pg_stats":[{"pgid":"2.f","version":"43'15966","reported_seq":16041,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.599958+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:18.599958+0000","last_peered":"2024-09-18T12:02:18.599958+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:18.599958+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14622720,"num_objects":3570,"num_object_clones":0,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15966,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-29}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15820","reported_seq":15981,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:18.629883+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:18.629883+0000","last_peered":"2024-09-18T12:02:18.629883+0000","last_clean":"2024-09-18T12:02:18.629883+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:18.629883+0000","last_undegraded":"2024-09-18T12:02:18.629883+0000","last_fullsized":"2024-09-18T12:02:18.629883+0000","mapping_epoch":29,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14794752,"num_objects":3612,"num_object_clones":0,"num_object_copies":7224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'15894","reported_seq":16084,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.684972+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:18.684972+0000","last_peered":"2024-09-18T12:02:18.684972+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:18.684972+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15003672,"num_objects":3664,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12752,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15708","reported_seq":15741,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:18.600549+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:18.600549+0000","last_peered":"2024-09-18T12:02:18.600549+0000","last_clean":"2024-09-18T12:02:18.600549+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:18.600549+0000","last_undegraded":"2024-09-18T12:02:18.600549+0000","last_fullsized":"2024-09-18T12:02:18.600549+0000","mapping_epoch":29,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14548992,"num_objects":3552,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15708,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15822","reported_seq":16017,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:18.625649+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:18.625649+0000","last_peered":"2024-09-18T12:02:18.625649+0000","last_clean":"2024-09-18T12:02:18.625649+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:18.625649+0000","last_undegraded":"2024-09-18T12:02:18.625649+0000","last_fullsized":"2024-09-18T12:02:18.625649+0000","mapping_epoch":29,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14848000,"num_objects":3625,"num_object_clones":0,"num_object_copies":7250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12987,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15786","reported_seq":15868,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.715601+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:18.715601+0000","last_peered":"2024-09-18T12:02:18.715601+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:18.715601+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14898386,"num_objects":3639,"num_object_clones":0,"num_object_copies":7278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15784,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":112}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15837","reported_seq":15928,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.668123+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:18.668123+0000","last_peered":"2024-09-18T12:02:18.668123+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:18.668123+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14480594,"num_objects":3537,"num_object_clones":0,"num_object_copies":7074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15835,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-22}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'15960","reported_seq":16188,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:18.695624+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:18.695624+0000","last_peered":"2024-09-18T12:02:18.695624+0000","last_clean":"2024-09-18T12:02:18.695624+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:18.695624+0000","last_undegraded":"2024-09-18T12:02:18.695624+0000","last_fullsized":"2024-09-18T12:02:18.695624+0000","mapping_epoch":29,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14917632,"num_objects":3642,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12277,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15819","reported_seq":22951,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:20.014522+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:20.014522+0000","last_peered":"2024-09-18T12:02:20.014522+0000","last_clean":"2024-09-18T12:02:20.014522+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:20.014522+0000","last_undegraded":"2024-09-18T12:02:20.014522+0000","last_fullsized":"2024-09-18T12:02:20.014522+0000","mapping_epoch":38,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14689490,"num_objects":3588,"num_object_clones":0,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15817,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'15889","reported_seq":16988,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.651644+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:18.651644+0000","last_peered":"2024-09-18T12:02:18.651644+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:18.651644+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14570706,"num_objects":3559,"num_object_clones":0,"num_object_copies":7118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15887,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":534}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'15906","reported_seq":15966,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.640881+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:18.640881+0000","last_peered":"2024-09-18T12:02:18.640881+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:18.640881+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14570706,"num_objects":3559,"num_object_clones":0,"num_object_copies":7118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15904,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":9}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'15883","reported_seq":23296,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:18.511522+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:18.511522+0000","last_peered":"2024-09-18T12:02:18.511522+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:18.511522+0000","last_undegraded":"2024-09-18T12:02:18.511522+0000","last_fullsized":"2024-09-18T12:02:18.511522+0000","mapping_epoch":42,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14807040,"num_objects":3615,"num_object_clones":0,"num_object_copies":7230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12363,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3615}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15831","reported_seq":15917,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.632902+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:18.632902+0000","last_peered":"2024-09-18T12:02:18.632902+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:18.632902+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15143529,"num_objects":3698,"num_object_clones":0,"num_object_copies":7396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15830,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":163}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'15986","reported_seq":23136,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:20.016003+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:20.016003+0000","last_peered":"2024-09-18T12:02:20.016003+0000","last_clean":"2024-09-18T12:02:20.016003+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:20.016003+0000","last_undegraded":"2024-09-18T12:02:20.016003+0000","last_fullsized":"2024-09-18T12:02:20.016003+0000","mapping_epoch":40,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14485307,"num_objects":3539,"num_object_clones":0,"num_object_copies":7078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15983,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15715","reported_seq":15800,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.639544+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:18.639544+0000","last_peered":"2024-09-18T12:02:18.639544+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:18.639544+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5700","ondisk_log_start":"37'5700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14267602,"num_objects":3485,"num_object_clones":0,"num_object_copies":6970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15713,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'15928","reported_seq":16017,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:18.647382+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:18.647382+0000","last_peered":"2024-09-18T12:02:18.647382+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:18.647382+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14682532,"num_objects":3588,"num_object_clones":0,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15925,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":235331660,"num_objects":57472,"num_object_clones":0,"num_object_copies":114944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237743,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11133,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":643747840,"data_stored":643684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051685,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527684,"kb_used_data":229816,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93844156,"statfs":{"total":96636764160,"available":96096415744,"internally_reserved":0,"allocated":235331584,"data_stored":235082949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.432}]}]},{"osd":1,"up_from":35,"seq":150323855448,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473896,"kb_used_data":171868,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93897944,"statfs":{"total":96636764160,"available":96151494656,"internally_reserved":0,"allocated":175992832,"data_stored":175695348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.44600000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.51400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539532,"kb_used_data":229244,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93832308,"statfs":{"total":96636764160,"available":96084283392,"internally_reserved":0,"allocated":234745856,"data_stored":234497671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:01:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6379999999999999,"5min":1.1160000000000001,"15min":1.0580000000000001},"min":{"1min":0.17599999999999999,"5min":0.161,"15min":0.161},"max":{"1min":8.1910000000000007,"5min":9.3200000000000003,"15min":9.3200000000000003},"last":0.503},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.93700000000000006,"15min":1.0860000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":4.7409999999999997,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.33500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.53600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234336256,"data_stored":234332777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174821376,"data_stored":174768598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":234590208,"data_stored":234582657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:24.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:25.751 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:25.769 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:25.788 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-09-18T12:02:25.227700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226340959,"num_objects":55279,"num_object_clones":0,"num_object_copies":110558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55279,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":239940,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11203,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160748,"ondisk_log_size":160748,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1523480,"kb_used_data":613292,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281592040,"statfs":{"total":289910292480,"available":288350248960,"internally_reserved":0,"allocated":628011008,"data_stored":627216704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,2,7,6,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27029504,"num_objects":-6599,"num_object_clones":0,"num_object_copies":-13198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6599,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005969"},"pg_stats":[{"pgid":"2.f","version":"43'16093","reported_seq":16168,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.563056+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:23.563056+0000","last_peered":"2024-09-18T12:02:23.563056+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:23.563056+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14102528,"num_objects":3443,"num_object_clones":0,"num_object_copies":6886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16093,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-156}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15974","reported_seq":16136,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:23.672951+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:23.672951+0000","last_peered":"2024-09-18T12:02:23.672951+0000","last_clean":"2024-09-18T12:02:23.672951+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:23.672951+0000","last_undegraded":"2024-09-18T12:02:23.672951+0000","last_fullsized":"2024-09-18T12:02:23.672951+0000","mapping_epoch":29,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14163968,"num_objects":3458,"num_object_clones":0,"num_object_copies":6916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13166,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16027","reported_seq":16214,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.624135+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:23.624135+0000","last_peered":"2024-09-18T12:02:23.624135+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:23.624135+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14458904,"num_objects":3531,"num_object_clones":0,"num_object_copies":7062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12885,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-5}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15862","reported_seq":15895,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:23.686434+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:23.686434+0000","last_peered":"2024-09-18T12:02:23.686434+0000","last_clean":"2024-09-18T12:02:23.686434+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:23.686434+0000","last_undegraded":"2024-09-18T12:02:23.686434+0000","last_fullsized":"2024-09-18T12:02:23.686434+0000","mapping_epoch":29,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13918208,"num_objects":3398,"num_object_clones":0,"num_object_copies":6796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15862,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15973","reported_seq":16168,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:23.666361+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:23.666361+0000","last_peered":"2024-09-18T12:02:23.666361+0000","last_clean":"2024-09-18T12:02:23.666361+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:23.666361+0000","last_undegraded":"2024-09-18T12:02:23.666361+0000","last_fullsized":"2024-09-18T12:02:23.666361+0000","mapping_epoch":29,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14229504,"num_objects":3474,"num_object_clones":0,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13138,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15916","reported_seq":16001,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.720400+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:23.720400+0000","last_peered":"2024-09-18T12:02:23.720400+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:23.720400+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14365906,"num_objects":3509,"num_object_clones":0,"num_object_copies":7018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15914,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-18}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15977","reported_seq":16068,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.669310+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:23.669310+0000","last_peered":"2024-09-18T12:02:23.669310+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:23.669310+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13907154,"num_objects":3397,"num_object_clones":0,"num_object_copies":6794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15975,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16123","reported_seq":16352,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:23.634465+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:23.634465+0000","last_peered":"2024-09-18T12:02:23.634465+0000","last_clean":"2024-09-18T12:02:23.634465+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:23.634465+0000","last_undegraded":"2024-09-18T12:02:23.634465+0000","last_fullsized":"2024-09-18T12:02:23.634465+0000","mapping_epoch":29,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14249984,"num_objects":3479,"num_object_clones":0,"num_object_copies":6958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12440,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15943","reported_seq":23075,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:25.004121+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:25.004121+0000","last_peered":"2024-09-18T12:02:25.004121+0000","last_clean":"2024-09-18T12:02:25.004121+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:25.004121+0000","last_undegraded":"2024-09-18T12:02:25.004121+0000","last_fullsized":"2024-09-18T12:02:25.004121+0000","mapping_epoch":38,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14181586,"num_objects":3464,"num_object_clones":0,"num_object_copies":6928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15941,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16023","reported_seq":17262,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.720488+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:23.720488+0000","last_peered":"2024-09-18T12:02:23.720488+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:23.720488+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14021842,"num_objects":3425,"num_object_clones":0,"num_object_copies":6850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16021,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":469}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16049","reported_seq":16109,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.626558+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:23.626558+0000","last_peered":"2024-09-18T12:02:23.626558+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:23.626558+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13984978,"num_objects":3416,"num_object_clones":0,"num_object_copies":6832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16047,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16017","reported_seq":23430,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:23.651521+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:23.651521+0000","last_peered":"2024-09-18T12:02:23.651521+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:23.651521+0000","last_undegraded":"2024-09-18T12:02:23.651521+0000","last_fullsized":"2024-09-18T12:02:23.651521+0000","mapping_epoch":42,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14258176,"num_objects":3481,"num_object_clones":0,"num_object_copies":6962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12497,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3481}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15953","reported_seq":16037,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.710122+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:23.710122+0000","last_peered":"2024-09-18T12:02:23.710122+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:23.710122+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14643817,"num_objects":3576,"num_object_clones":0,"num_object_copies":7152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15952,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":41}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16116","reported_seq":23266,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:25.032508+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:25.032508+0000","last_peered":"2024-09-18T12:02:25.032508+0000","last_clean":"2024-09-18T12:02:25.032508+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:25.032508+0000","last_undegraded":"2024-09-18T12:02:25.032508+0000","last_fullsized":"2024-09-18T12:02:25.032508+0000","mapping_epoch":40,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13952827,"num_objects":3409,"num_object_clones":0,"num_object_copies":6818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16113,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15833","reported_seq":15917,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.724675+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:23.724675+0000","last_peered":"2024-09-18T12:02:23.724675+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:23.724675+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13784274,"num_objects":3367,"num_object_clones":0,"num_object_copies":6734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15831,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16066","reported_seq":16155,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.686281+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:23.686281+0000","last_peered":"2024-09-18T12:02:23.686281+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:23.686281+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14117284,"num_objects":3450,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16063,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-73}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":226340940,"num_objects":55277,"num_object_clones":0,"num_object_copies":110554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55277,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":239938,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11202,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":625631232,"data_stored":625567424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160745,"ondisk_log_size":160745,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051686,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527160,"kb_used_data":225120,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93844680,"statfs":{"total":96636764160,"available":96096952320,"internally_reserved":0,"allocated":230522880,"data_stored":230274245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,3,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.18099999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":4.9619999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.23200000000000001},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.33500000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855449,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471736,"kb_used_data":165548,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93900104,"statfs":{"total":96636764160,"available":96153706496,"internally_reserved":0,"allocated":169521152,"data_stored":169223668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.185}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.35099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524584,"kb_used_data":222624,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93847256,"statfs":{"total":96636764160,"available":96099590144,"internally_reserved":0,"allocated":227966976,"data_stored":227718791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":2.3620000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.374}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":2.2370000000000001},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":2.403}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227557376,"data_stored":227553897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168226816,"data_stored":168174038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229847040,"data_stored":229839489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:25.789 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:02:25.789 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'16093", 'reported_seq': 16168, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:23.563056+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:02:23.563056+0000', 'last_peered': '2024-09-18T12:02:23.563056+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:02:23.563056+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'6000", 'ondisk_log_start': "37'6000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14102528, 'num_objects': 3443, 'num_object_clones': 0, 'num_object_copies': 6886, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16093, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -156}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:25.789 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:02:25.789 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'16027", 'reported_seq': 16214, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:23.624135+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:02:23.624135+0000', 'last_peered': '2024-09-18T12:02:23.624135+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:02:23.624135+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'6000", 'ondisk_log_start': "37'6000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14458904, 'num_objects': 3531, 'num_object_clones': 0, 'num_object_copies': 7062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3531, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12885, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -5}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:25.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:02:25.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'15916", 'reported_seq': 16001, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:23.720400+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:02:23.720400+0000', 'last_peered': '2024-09-18T12:02:23.720400+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:02:23.720400+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'5900", 'ondisk_log_start': "37'5900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14365906, 'num_objects': 3509, 'num_object_clones': 0, 'num_object_copies': 7018, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15914, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -18}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:25.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:02:25.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'15977", 'reported_seq': 16068, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:23.669310+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:02:23.669310+0000', 'last_peered': '2024-09-18T12:02:23.669310+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:02:23.669310+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'5900", 'ondisk_log_start': "37'5900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13907154, 'num_objects': 3397, 'num_object_clones': 0, 'num_object_copies': 6794, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15975, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:25.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:02:25.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'16023", 'reported_seq': 17262, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:23.720488+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:02:23.720488+0000', 'last_peered': '2024-09-18T12:02:23.720488+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:02:23.720488+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'6000", 'ondisk_log_start': "37'6000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14021842, 'num_objects': 3425, 'num_object_clones': 0, 'num_object_copies': 6850, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2956, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16021, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 580, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 469}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:25.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:02:25.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'16049", 'reported_seq': 16109, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:23.626558+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:02:23.626558+0000', 'last_peered': '2024-09-18T12:02:23.626558+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:02:23.626558+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'6000", 'ondisk_log_start': "37'6000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13984978, 'num_objects': 3416, 'num_object_clones': 0, 'num_object_copies': 6832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16047, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -134}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:25.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:02:25.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'16017", 'reported_seq': 23430, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:02:23.651521+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:02:23.651521+0000', 'last_peered': '2024-09-18T12:02:23.651521+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:02:23.651521+0000', 'last_undegraded': '2024-09-18T12:02:23.651521+0000', 'last_fullsized': '2024-09-18T12:02:23.651521+0000', 'mapping_epoch': 42, 'log_start': "37'6000", 'ondisk_log_start': "37'6000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14258176, 'num_objects': 3481, 'num_object_clones': 0, 'num_object_copies': 6962, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12497, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 3481}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:25.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:02:25.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'15953", 'reported_seq': 16037, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:23.710122+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:02:23.710122+0000', 'last_peered': '2024-09-18T12:02:23.710122+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:02:23.710122+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'5900", 'ondisk_log_start': "37'5900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14643817, 'num_objects': 3576, 'num_object_clones': 0, 'num_object_copies': 7152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15952, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 41}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:25.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:02:25.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'15833", 'reported_seq': 15917, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:23.724675+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:02:23.724675+0000', 'last_peered': '2024-09-18T12:02:23.724675+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:02:23.724675+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'5800", 'ondisk_log_start': "37'5800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13784274, 'num_objects': 3367, 'num_object_clones': 0, 'num_object_copies': 6734, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15831, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:25.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:02:25.791 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'16066", 'reported_seq': 16155, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:23.686281+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:02:23.686281+0000', 'last_peered': '2024-09-18T12:02:23.686281+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:02:23.686281+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'6000", 'ondisk_log_start': "37'6000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14117284, 'num_objects': 3450, 'num_object_clones': 0, 'num_object_copies': 6900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16063, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -73}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:25.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:02:26.784 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:26.784 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31391/110558 objects degraded (28.393%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":546,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":55279,"data_bytes":226340959,"bytes_used":1560043520,"bytes_avail":288350248960,"bytes_total":289910292480,"degraded_objects":31391,"degraded_total":110558,"degraded_ratio":0.28393241556468096,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":208,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":549},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:02:26.823 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:02:26.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:27.406 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:27.422 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:27.431 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-18T12:02:27.228172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226340959,"num_objects":55279,"num_object_clones":0,"num_object_copies":110558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55279,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":239940,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11203,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160748,"ondisk_log_size":160748,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1523480,"kb_used_data":613292,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281592040,"statfs":{"total":289910292480,"available":288350248960,"internally_reserved":0,"allocated":628011008,"data_stored":627216704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,2,7,6,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18067456,"num_objects":-4411,"num_object_clones":0,"num_object_copies":-8822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4411,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006009"},"pg_stats":[{"pgid":"2.f","version":"43'16093","reported_seq":16168,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.563056+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:23.563056+0000","last_peered":"2024-09-18T12:02:23.563056+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:23.563056+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14102528,"num_objects":3443,"num_object_clones":0,"num_object_copies":6886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16093,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-156}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'15974","reported_seq":16136,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:23.672951+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:23.672951+0000","last_peered":"2024-09-18T12:02:23.672951+0000","last_clean":"2024-09-18T12:02:23.672951+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:23.672951+0000","last_undegraded":"2024-09-18T12:02:23.672951+0000","last_fullsized":"2024-09-18T12:02:23.672951+0000","mapping_epoch":29,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14163968,"num_objects":3458,"num_object_clones":0,"num_object_copies":6916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13166,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16027","reported_seq":16214,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.624135+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:23.624135+0000","last_peered":"2024-09-18T12:02:23.624135+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:23.624135+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14458904,"num_objects":3531,"num_object_clones":0,"num_object_copies":7062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3531,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12885,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-5}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'15862","reported_seq":15895,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:23.686434+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:23.686434+0000","last_peered":"2024-09-18T12:02:23.686434+0000","last_clean":"2024-09-18T12:02:23.686434+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:23.686434+0000","last_undegraded":"2024-09-18T12:02:23.686434+0000","last_fullsized":"2024-09-18T12:02:23.686434+0000","mapping_epoch":29,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13918208,"num_objects":3398,"num_object_clones":0,"num_object_copies":6796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15862,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'15973","reported_seq":16168,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:23.666361+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:23.666361+0000","last_peered":"2024-09-18T12:02:23.666361+0000","last_clean":"2024-09-18T12:02:23.666361+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:23.666361+0000","last_undegraded":"2024-09-18T12:02:23.666361+0000","last_fullsized":"2024-09-18T12:02:23.666361+0000","mapping_epoch":29,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14229504,"num_objects":3474,"num_object_clones":0,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13138,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'15916","reported_seq":16001,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.720400+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:23.720400+0000","last_peered":"2024-09-18T12:02:23.720400+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:23.720400+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14365906,"num_objects":3509,"num_object_clones":0,"num_object_copies":7018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15914,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-18}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'15977","reported_seq":16068,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.669310+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:23.669310+0000","last_peered":"2024-09-18T12:02:23.669310+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:23.669310+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13907154,"num_objects":3397,"num_object_clones":0,"num_object_copies":6794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15975,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16123","reported_seq":16352,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:23.634465+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:23.634465+0000","last_peered":"2024-09-18T12:02:23.634465+0000","last_clean":"2024-09-18T12:02:23.634465+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:23.634465+0000","last_undegraded":"2024-09-18T12:02:23.634465+0000","last_fullsized":"2024-09-18T12:02:23.634465+0000","mapping_epoch":29,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14249984,"num_objects":3479,"num_object_clones":0,"num_object_copies":6958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12440,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15943","reported_seq":23075,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:25.004121+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:25.004121+0000","last_peered":"2024-09-18T12:02:25.004121+0000","last_clean":"2024-09-18T12:02:25.004121+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:25.004121+0000","last_undegraded":"2024-09-18T12:02:25.004121+0000","last_fullsized":"2024-09-18T12:02:25.004121+0000","mapping_epoch":38,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14181586,"num_objects":3464,"num_object_clones":0,"num_object_copies":6928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15941,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16023","reported_seq":17262,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.720488+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:23.720488+0000","last_peered":"2024-09-18T12:02:23.720488+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:23.720488+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14021842,"num_objects":3425,"num_object_clones":0,"num_object_copies":6850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16021,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":469}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16049","reported_seq":16109,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.626558+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:23.626558+0000","last_peered":"2024-09-18T12:02:23.626558+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:23.626558+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13984978,"num_objects":3416,"num_object_clones":0,"num_object_copies":6832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16047,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16017","reported_seq":23430,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:23.651521+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:23.651521+0000","last_peered":"2024-09-18T12:02:23.651521+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:23.651521+0000","last_undegraded":"2024-09-18T12:02:23.651521+0000","last_fullsized":"2024-09-18T12:02:23.651521+0000","mapping_epoch":42,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14258176,"num_objects":3481,"num_object_clones":0,"num_object_copies":6962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12497,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3481}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'15953","reported_seq":16037,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.710122+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:23.710122+0000","last_peered":"2024-09-18T12:02:23.710122+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:23.710122+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14643817,"num_objects":3576,"num_object_clones":0,"num_object_copies":7152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15952,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":41}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16116","reported_seq":23266,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:25.032508+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:25.032508+0000","last_peered":"2024-09-18T12:02:25.032508+0000","last_clean":"2024-09-18T12:02:25.032508+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:25.032508+0000","last_undegraded":"2024-09-18T12:02:25.032508+0000","last_fullsized":"2024-09-18T12:02:25.032508+0000","mapping_epoch":40,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13952827,"num_objects":3409,"num_object_clones":0,"num_object_copies":6818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16113,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15833","reported_seq":15917,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.724675+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:23.724675+0000","last_peered":"2024-09-18T12:02:23.724675+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:23.724675+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5800","ondisk_log_start":"37'5800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13784274,"num_objects":3367,"num_object_clones":0,"num_object_copies":6734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15831,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16066","reported_seq":16155,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:23.686281+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:23.686281+0000","last_peered":"2024-09-18T12:02:23.686281+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:23.686281+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14117284,"num_objects":3450,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16063,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-73}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":226340940,"num_objects":55277,"num_object_clones":0,"num_object_copies":110554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55277,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":239938,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11202,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":625631232,"data_stored":625567424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160745,"ondisk_log_size":160745,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051686,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527160,"kb_used_data":225120,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93844680,"statfs":{"total":96636764160,"available":96096952320,"internally_reserved":0,"allocated":230522880,"data_stored":230274245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,3,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:01:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.80100000000000005,"15min":0.54200000000000004},"min":{"1min":0.216,"5min":0.153,"15min":0.153},"max":{"1min":7.1740000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.18099999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.385,"15min":0.75700000000000001},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":6.1420000000000003,"5min":10.795999999999999,"15min":10.795999999999999},"last":4.9619999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.23200000000000001},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.33500000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855449,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471736,"kb_used_data":165548,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93900104,"statfs":{"total":96636764160,"available":96153706496,"internally_reserved":0,"allocated":169521152,"data_stored":169223668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.185}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.35099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524584,"kb_used_data":222624,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93847256,"statfs":{"total":96636764160,"available":96099590144,"internally_reserved":0,"allocated":227966976,"data_stored":227718791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":2.3620000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.374}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":2.2370000000000001},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":2.403}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227557376,"data_stored":227553897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168226816,"data_stored":168174038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229847040,"data_stored":229839489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:30.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:31.140 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:31.166 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:31.187 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-18T12:02:29.228576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":218579039,"num_objects":53384,"num_object_clones":0,"num_object_copies":106768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53384,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":241835,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11277,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160843,"ondisk_log_size":160843,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1520992,"kb_used_data":602492,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281594528,"statfs":{"total":289910292480,"available":288352796672,"internally_reserved":0,"allocated":616951808,"data_stored":616157504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,6,7,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25829376,"num_objects":-6306,"num_object_clones":0,"num_object_copies":-12612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6306,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005063"},"pg_stats":[{"pgid":"2.f","version":"43'16229","reported_seq":16304,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.715412+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:28.715412+0000","last_peered":"2024-09-18T12:02:28.715412+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:28.715412+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13545472,"num_objects":3307,"num_object_clones":0,"num_object_copies":6614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16229,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-292}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16122","reported_seq":16280,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:28.658600+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:28.658600+0000","last_peered":"2024-09-18T12:02:28.658600+0000","last_clean":"2024-09-18T12:02:28.658600+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:28.658600+0000","last_undegraded":"2024-09-18T12:02:28.658600+0000","last_fullsized":"2024-09-18T12:02:28.658600+0000","mapping_epoch":29,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13557760,"num_objects":3310,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13314,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16156","reported_seq":16346,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.594394+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:28.594394+0000","last_peered":"2024-09-18T12:02:28.594394+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:28.594394+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13930520,"num_objects":3402,"num_object_clones":0,"num_object_copies":6804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13014,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-134}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16000","reported_seq":16033,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:28.632693+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:28.632693+0000","last_peered":"2024-09-18T12:02:28.632693+0000","last_clean":"2024-09-18T12:02:28.632693+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:28.632693+0000","last_undegraded":"2024-09-18T12:02:28.632693+0000","last_fullsized":"2024-09-18T12:02:28.632693+0000","mapping_epoch":29,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13352960,"num_objects":3260,"num_object_clones":0,"num_object_copies":6520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16000,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16123","reported_seq":16314,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:28.719684+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:28.719684+0000","last_peered":"2024-09-18T12:02:28.719684+0000","last_clean":"2024-09-18T12:02:28.719684+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:28.719684+0000","last_undegraded":"2024-09-18T12:02:28.719684+0000","last_fullsized":"2024-09-18T12:02:28.719684+0000","mapping_epoch":29,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13615104,"num_objects":3324,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16047","reported_seq":16132,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.669746+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:28.669746+0000","last_peered":"2024-09-18T12:02:28.669746+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:28.669746+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13829330,"num_objects":3378,"num_object_clones":0,"num_object_copies":6756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16045,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16110","reported_seq":16201,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.590383+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:28.590383+0000","last_peered":"2024-09-18T12:02:28.590383+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:28.590383+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13362386,"num_objects":3264,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16108,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-295}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16278","reported_seq":16507,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:28.755684+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:28.755684+0000","last_peered":"2024-09-18T12:02:28.755684+0000","last_clean":"2024-09-18T12:02:28.755684+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:28.755684+0000","last_undegraded":"2024-09-18T12:02:28.755684+0000","last_fullsized":"2024-09-18T12:02:28.755684+0000","mapping_epoch":29,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13615104,"num_objects":3324,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12595,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'15943","reported_seq":23075,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:25.004121+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:25.004121+0000","last_peered":"2024-09-18T12:02:25.004121+0000","last_clean":"2024-09-18T12:02:25.004121+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:25.004121+0000","last_undegraded":"2024-09-18T12:02:25.004121+0000","last_fullsized":"2024-09-18T12:02:25.004121+0000","mapping_epoch":38,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14181586,"num_objects":3464,"num_object_clones":0,"num_object_copies":6928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15941,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16158","reported_seq":17545,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.697145+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:28.697145+0000","last_peered":"2024-09-18T12:02:28.697145+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:28.697145+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13468882,"num_objects":3290,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16156,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":654,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":408}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16164","reported_seq":16224,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.530734+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:28.530734+0000","last_peered":"2024-09-18T12:02:28.530734+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:28.530734+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13513938,"num_objects":3301,"num_object_clones":0,"num_object_copies":6602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16162,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-249}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16159","reported_seq":23572,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:28.679515+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:28.679515+0000","last_peered":"2024-09-18T12:02:28.679515+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:28.679515+0000","last_undegraded":"2024-09-18T12:02:28.679515+0000","last_fullsized":"2024-09-18T12:02:28.679515+0000","mapping_epoch":42,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13676544,"num_objects":3339,"num_object_clones":0,"num_object_copies":6678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12639,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16072","reported_seq":16158,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.715416+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:28.715416+0000","last_peered":"2024-09-18T12:02:28.715416+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:28.715416+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14156393,"num_objects":3457,"num_object_clones":0,"num_object_copies":6914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16071,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16116","reported_seq":23266,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:25.032508+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:25.032508+0000","last_peered":"2024-09-18T12:02:25.032508+0000","last_clean":"2024-09-18T12:02:25.032508+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:25.032508+0000","last_undegraded":"2024-09-18T12:02:25.032508+0000","last_fullsized":"2024-09-18T12:02:25.032508+0000","mapping_epoch":40,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13952827,"num_objects":3409,"num_object_clones":0,"num_object_copies":6818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16113,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15969","reported_seq":16054,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.684867+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:28.684867+0000","last_peered":"2024-09-18T12:02:28.684867+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:28.684867+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227218,"num_objects":3231,"num_object_clones":0,"num_object_copies":6462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15967,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16194","reported_seq":16283,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.696462+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:28.696462+0000","last_peered":"2024-09-18T12:02:28.696462+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:28.696462+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13592996,"num_objects":3322,"num_object_clones":0,"num_object_copies":6644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16191,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":218579020,"num_objects":53382,"num_object_clones":0,"num_object_copies":106764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":241833,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11276,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":614281216,"data_stored":614217408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160840,"ondisk_log_size":160840,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051687,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526940,"kb_used_data":220752,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93844900,"statfs":{"total":96636764160,"available":96097177600,"internally_reserved":0,"allocated":226050048,"data_stored":225801413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.26000000000000001},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":9.0530000000000008}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":9.1050000000000004},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.22700000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855449,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471736,"kb_used_data":165548,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93900104,"statfs":{"total":96636764160,"available":96153706496,"internally_reserved":0,"allocated":169521152,"data_stored":169223668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.185}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.35099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522316,"kb_used_data":216192,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93849524,"statfs":{"total":96636764160,"available":96101912576,"internally_reserved":0,"allocated":221380608,"data_stored":221132423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.29099999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220782592,"data_stored":220779113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168226816,"data_stored":168174038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225271808,"data_stored":225264257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:31.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:31.932 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:31.956 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:31.972 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-09-18T12:02:31.229030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":217272415,"num_objects":53065,"num_object_clones":0,"num_object_copies":106130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242154,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11277,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160862,"ondisk_log_size":160862,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1506320,"kb_used_data":596140,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281609200,"statfs":{"total":289910292480,"available":288367820800,"internally_reserved":0,"allocated":610447360,"data_stored":609653056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,6,9,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19177472,"num_objects":-4682,"num_object_clones":0,"num_object_copies":-9364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4682,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005087"},"pg_stats":[{"pgid":"2.f","version":"43'16229","reported_seq":16304,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.715412+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:28.715412+0000","last_peered":"2024-09-18T12:02:28.715412+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:28.715412+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13545472,"num_objects":3307,"num_object_clones":0,"num_object_copies":6614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16229,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-292}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16122","reported_seq":16280,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:28.658600+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:28.658600+0000","last_peered":"2024-09-18T12:02:28.658600+0000","last_clean":"2024-09-18T12:02:28.658600+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:28.658600+0000","last_undegraded":"2024-09-18T12:02:28.658600+0000","last_fullsized":"2024-09-18T12:02:28.658600+0000","mapping_epoch":29,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13557760,"num_objects":3310,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13314,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16156","reported_seq":16346,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.594394+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:28.594394+0000","last_peered":"2024-09-18T12:02:28.594394+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:28.594394+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13930520,"num_objects":3402,"num_object_clones":0,"num_object_copies":6804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13014,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-134}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16000","reported_seq":16033,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:28.632693+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:28.632693+0000","last_peered":"2024-09-18T12:02:28.632693+0000","last_clean":"2024-09-18T12:02:28.632693+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:28.632693+0000","last_undegraded":"2024-09-18T12:02:28.632693+0000","last_fullsized":"2024-09-18T12:02:28.632693+0000","mapping_epoch":29,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13352960,"num_objects":3260,"num_object_clones":0,"num_object_copies":6520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16000,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16123","reported_seq":16314,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:28.719684+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:28.719684+0000","last_peered":"2024-09-18T12:02:28.719684+0000","last_clean":"2024-09-18T12:02:28.719684+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:28.719684+0000","last_undegraded":"2024-09-18T12:02:28.719684+0000","last_fullsized":"2024-09-18T12:02:28.719684+0000","mapping_epoch":29,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13615104,"num_objects":3324,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16047","reported_seq":16132,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.669746+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:28.669746+0000","last_peered":"2024-09-18T12:02:28.669746+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:28.669746+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13829330,"num_objects":3378,"num_object_clones":0,"num_object_copies":6756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16045,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16110","reported_seq":16201,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.590383+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:28.590383+0000","last_peered":"2024-09-18T12:02:28.590383+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:28.590383+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13362386,"num_objects":3264,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16108,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-295}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16278","reported_seq":16507,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:28.755684+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:28.755684+0000","last_peered":"2024-09-18T12:02:28.755684+0000","last_clean":"2024-09-18T12:02:28.755684+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:28.755684+0000","last_undegraded":"2024-09-18T12:02:28.755684+0000","last_fullsized":"2024-09-18T12:02:28.755684+0000","mapping_epoch":29,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13615104,"num_objects":3324,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12595,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16107","reported_seq":23239,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:30.124968+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:30.124968+0000","last_peered":"2024-09-18T12:02:30.124968+0000","last_clean":"2024-09-18T12:02:30.124968+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:30.124968+0000","last_undegraded":"2024-09-18T12:02:30.124968+0000","last_fullsized":"2024-09-18T12:02:30.124968+0000","mapping_epoch":38,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13509842,"num_objects":3300,"num_object_clones":0,"num_object_copies":6600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16105,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16158","reported_seq":17545,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.697145+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:28.697145+0000","last_peered":"2024-09-18T12:02:28.697145+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:28.697145+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13468882,"num_objects":3290,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16156,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":654,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":408}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16164","reported_seq":16224,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.530734+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:28.530734+0000","last_peered":"2024-09-18T12:02:28.530734+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:28.530734+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13513938,"num_objects":3301,"num_object_clones":0,"num_object_copies":6602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16162,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-249}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16159","reported_seq":23572,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:28.679515+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:28.679515+0000","last_peered":"2024-09-18T12:02:28.679515+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:28.679515+0000","last_undegraded":"2024-09-18T12:02:28.679515+0000","last_fullsized":"2024-09-18T12:02:28.679515+0000","mapping_epoch":42,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13676544,"num_objects":3339,"num_object_clones":0,"num_object_copies":6678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12639,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16072","reported_seq":16158,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.715416+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:28.715416+0000","last_peered":"2024-09-18T12:02:28.715416+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:28.715416+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14156393,"num_objects":3457,"num_object_clones":0,"num_object_copies":6914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16071,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16271","reported_seq":23421,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:30.013665+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:30.013665+0000","last_peered":"2024-09-18T12:02:30.013665+0000","last_clean":"2024-09-18T12:02:30.013665+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:30.013665+0000","last_undegraded":"2024-09-18T12:02:30.013665+0000","last_fullsized":"2024-09-18T12:02:30.013665+0000","mapping_epoch":40,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13317947,"num_objects":3254,"num_object_clones":0,"num_object_copies":6508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16268,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15969","reported_seq":16054,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.684867+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:28.684867+0000","last_peered":"2024-09-18T12:02:28.684867+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:28.684867+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227218,"num_objects":3231,"num_object_clones":0,"num_object_copies":6462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15967,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16194","reported_seq":16283,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.696462+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:28.696462+0000","last_peered":"2024-09-18T12:02:28.696462+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:28.696462+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13592996,"num_objects":3322,"num_object_clones":0,"num_object_copies":6644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16191,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":217272396,"num_objects":53063,"num_object_clones":0,"num_object_copies":106126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53063,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242152,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11276,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":607682560,"data_stored":607618752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051687,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526940,"kb_used_data":220752,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93844900,"statfs":{"total":96636764160,"available":96097177600,"internally_reserved":0,"allocated":226050048,"data_stored":225801413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.26000000000000001},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":9.0530000000000008}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":9.1050000000000004},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.22700000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855450,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457064,"kb_used_data":159196,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93914776,"statfs":{"total":96636764160,"available":96168730624,"internally_reserved":0,"allocated":163016704,"data_stored":162719220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.247}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.35499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522316,"kb_used_data":216192,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93849524,"statfs":{"total":96636764160,"available":96101912576,"internally_reserved":0,"allocated":221380608,"data_stored":221132423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.29099999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220782592,"data_stored":220779113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161628160,"data_stored":161575382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225271808,"data_stored":225264257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:31.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:02:31.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'16229", 'reported_seq': 16304, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:28.715412+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:02:28.715412+0000', 'last_peered': '2024-09-18T12:02:28.715412+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:02:28.715412+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'6200", 'ondisk_log_start': "37'6200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13545472, 'num_objects': 3307, 'num_object_clones': 0, 'num_object_copies': 6614, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16229, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -292}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:31.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:02:31.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'16156", 'reported_seq': 16346, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:28.594394+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:02:28.594394+0000', 'last_peered': '2024-09-18T12:02:28.594394+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:02:28.594394+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'6100", 'ondisk_log_start': "37'6100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13930520, 'num_objects': 3402, 'num_object_clones': 0, 'num_object_copies': 6804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3402, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13014, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:31.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:02:31.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'16047", 'reported_seq': 16132, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:28.669746+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:02:28.669746+0000', 'last_peered': '2024-09-18T12:02:28.669746+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:02:28.669746+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'6000", 'ondisk_log_start': "37'6000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13829330, 'num_objects': 3378, 'num_object_clones': 0, 'num_object_copies': 6756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16045, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -149}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:31.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:02:31.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'16110", 'reported_seq': 16201, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:28.590383+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:02:28.590383+0000', 'last_peered': '2024-09-18T12:02:28.590383+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:02:28.590383+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'6100", 'ondisk_log_start': "37'6100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13362386, 'num_objects': 3264, 'num_object_clones': 0, 'num_object_copies': 6528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16108, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -295}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:31.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:02:31.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'16158", 'reported_seq': 17545, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:28.697145+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:02:28.697145+0000', 'last_peered': '2024-09-18T12:02:28.697145+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:02:28.697145+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'6100", 'ondisk_log_start': "37'6100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13468882, 'num_objects': 3290, 'num_object_clones': 0, 'num_object_copies': 6580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2882, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16156, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 654, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 408}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:31.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:02:31.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'16164", 'reported_seq': 16224, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:28.530734+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:02:28.530734+0000', 'last_peered': '2024-09-18T12:02:28.530734+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:02:28.530734+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'6100", 'ondisk_log_start': "37'6100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13513938, 'num_objects': 3301, 'num_object_clones': 0, 'num_object_copies': 6602, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16162, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -249}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:31.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:02:31.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'16159", 'reported_seq': 23572, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:02:28.679515+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:02:28.679515+0000', 'last_peered': '2024-09-18T12:02:28.679515+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:02:28.679515+0000', 'last_undegraded': '2024-09-18T12:02:28.679515+0000', 'last_fullsized': '2024-09-18T12:02:28.679515+0000', 'mapping_epoch': 42, 'log_start': "37'6100", 'ondisk_log_start': "37'6100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13676544, 'num_objects': 3339, 'num_object_clones': 0, 'num_object_copies': 6678, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12639, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 3339}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:31.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:02:31.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'16072", 'reported_seq': 16158, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:28.715416+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:02:28.715416+0000', 'last_peered': '2024-09-18T12:02:28.715416+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:02:28.715416+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'6000", 'ondisk_log_start': "37'6000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14156393, 'num_objects': 3457, 'num_object_clones': 0, 'num_object_copies': 6914, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16071, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -78}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:31.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:02:31.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'15969", 'reported_seq': 16054, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:28.684867+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:02:28.684867+0000', 'last_peered': '2024-09-18T12:02:28.684867+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:02:28.684867+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'5900", 'ondisk_log_start': "37'5900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13227218, 'num_objects': 3231, 'num_object_clones': 0, 'num_object_copies': 6462, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3231, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15967, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -362}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:31.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:02:31.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'16194", 'reported_seq': 16283, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:28.696462+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:02:28.696462+0000', 'last_peered': '2024-09-18T12:02:28.696462+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:02:28.696462+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'6100", 'ondisk_log_start': "37'6100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13592996, 'num_objects': 3322, 'num_object_clones': 0, 'num_object_copies': 6644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16191, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:31.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:02:32.703 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:32.703 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31317/106130 objects degraded (29.508%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":552,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":53065,"data_bytes":217272415,"bytes_used":1542471680,"bytes_avail":288367820800,"bytes_total":289910292480,"degraded_objects":31317,"degraded_total":106130,"degraded_ratio":0.29508150381607462,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":390},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:02:32.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:02:32.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:33.475 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:33.487 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:33.504 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-18T12:02:33.229485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":217272415,"num_objects":53065,"num_object_clones":0,"num_object_copies":106130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242154,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11277,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160862,"ondisk_log_size":160862,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1506320,"kb_used_data":596140,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281609200,"statfs":{"total":289910292480,"available":288367820800,"internally_reserved":0,"allocated":610447360,"data_stored":609653056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,6,9,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18059264,"num_objects":-4409,"num_object_clones":0,"num_object_copies":-8818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4409,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002690"},"pg_stats":[{"pgid":"2.f","version":"43'16229","reported_seq":16304,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.715412+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:28.715412+0000","last_peered":"2024-09-18T12:02:28.715412+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:28.715412+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13545472,"num_objects":3307,"num_object_clones":0,"num_object_copies":6614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16229,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-292}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16122","reported_seq":16280,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:28.658600+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:28.658600+0000","last_peered":"2024-09-18T12:02:28.658600+0000","last_clean":"2024-09-18T12:02:28.658600+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:28.658600+0000","last_undegraded":"2024-09-18T12:02:28.658600+0000","last_fullsized":"2024-09-18T12:02:28.658600+0000","mapping_epoch":29,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13557760,"num_objects":3310,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13314,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16156","reported_seq":16346,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.594394+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:28.594394+0000","last_peered":"2024-09-18T12:02:28.594394+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:28.594394+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13930520,"num_objects":3402,"num_object_clones":0,"num_object_copies":6804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3402,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13014,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-134}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16000","reported_seq":16033,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:28.632693+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:28.632693+0000","last_peered":"2024-09-18T12:02:28.632693+0000","last_clean":"2024-09-18T12:02:28.632693+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:28.632693+0000","last_undegraded":"2024-09-18T12:02:28.632693+0000","last_fullsized":"2024-09-18T12:02:28.632693+0000","mapping_epoch":29,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13352960,"num_objects":3260,"num_object_clones":0,"num_object_copies":6520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16000,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16123","reported_seq":16314,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:28.719684+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:28.719684+0000","last_peered":"2024-09-18T12:02:28.719684+0000","last_clean":"2024-09-18T12:02:28.719684+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:28.719684+0000","last_undegraded":"2024-09-18T12:02:28.719684+0000","last_fullsized":"2024-09-18T12:02:28.719684+0000","mapping_epoch":29,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13615104,"num_objects":3324,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16047","reported_seq":16132,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.669746+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:28.669746+0000","last_peered":"2024-09-18T12:02:28.669746+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:28.669746+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13829330,"num_objects":3378,"num_object_clones":0,"num_object_copies":6756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16045,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-149}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16110","reported_seq":16201,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.590383+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:28.590383+0000","last_peered":"2024-09-18T12:02:28.590383+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:28.590383+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13362386,"num_objects":3264,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16108,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-295}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16278","reported_seq":16507,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:28.755684+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:28.755684+0000","last_peered":"2024-09-18T12:02:28.755684+0000","last_clean":"2024-09-18T12:02:28.755684+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:28.755684+0000","last_undegraded":"2024-09-18T12:02:28.755684+0000","last_fullsized":"2024-09-18T12:02:28.755684+0000","mapping_epoch":29,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13615104,"num_objects":3324,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12595,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16107","reported_seq":23239,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:30.124968+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:30.124968+0000","last_peered":"2024-09-18T12:02:30.124968+0000","last_clean":"2024-09-18T12:02:30.124968+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:30.124968+0000","last_undegraded":"2024-09-18T12:02:30.124968+0000","last_fullsized":"2024-09-18T12:02:30.124968+0000","mapping_epoch":38,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13509842,"num_objects":3300,"num_object_clones":0,"num_object_copies":6600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16105,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16158","reported_seq":17545,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.697145+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:28.697145+0000","last_peered":"2024-09-18T12:02:28.697145+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:28.697145+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13468882,"num_objects":3290,"num_object_clones":0,"num_object_copies":6580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16156,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":654,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":408}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16164","reported_seq":16224,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.530734+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:28.530734+0000","last_peered":"2024-09-18T12:02:28.530734+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:28.530734+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13513938,"num_objects":3301,"num_object_clones":0,"num_object_copies":6602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16162,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-249}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16159","reported_seq":23572,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:28.679515+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:28.679515+0000","last_peered":"2024-09-18T12:02:28.679515+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:28.679515+0000","last_undegraded":"2024-09-18T12:02:28.679515+0000","last_fullsized":"2024-09-18T12:02:28.679515+0000","mapping_epoch":42,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13676544,"num_objects":3339,"num_object_clones":0,"num_object_copies":6678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12639,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16072","reported_seq":16158,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.715416+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:28.715416+0000","last_peered":"2024-09-18T12:02:28.715416+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:28.715416+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14156393,"num_objects":3457,"num_object_clones":0,"num_object_copies":6914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16071,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16271","reported_seq":23421,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:30.013665+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:30.013665+0000","last_peered":"2024-09-18T12:02:30.013665+0000","last_clean":"2024-09-18T12:02:30.013665+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:30.013665+0000","last_undegraded":"2024-09-18T12:02:30.013665+0000","last_fullsized":"2024-09-18T12:02:30.013665+0000","mapping_epoch":40,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13317947,"num_objects":3254,"num_object_clones":0,"num_object_copies":6508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16268,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'15969","reported_seq":16054,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.684867+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:28.684867+0000","last_peered":"2024-09-18T12:02:28.684867+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:28.684867+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'5900","ondisk_log_start":"37'5900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227218,"num_objects":3231,"num_object_clones":0,"num_object_copies":6462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15967,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-362}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16194","reported_seq":16283,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:28.696462+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:28.696462+0000","last_peered":"2024-09-18T12:02:28.696462+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:28.696462+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13592996,"num_objects":3322,"num_object_clones":0,"num_object_copies":6644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16191,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":217272396,"num_objects":53063,"num_object_clones":0,"num_object_copies":106126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53063,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242152,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11276,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":607682560,"data_stored":607618752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160859,"ondisk_log_size":160859,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051687,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526940,"kb_used_data":220752,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93844900,"statfs":{"total":96636764160,"available":96097177600,"internally_reserved":0,"allocated":226050048,"data_stored":225801413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.26000000000000001},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":9.0530000000000008}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":9.1050000000000004},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.22700000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855450,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457064,"kb_used_data":159196,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93914776,"statfs":{"total":96636764160,"available":96168730624,"internally_reserved":0,"allocated":163016704,"data_stored":162719220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.247}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.35499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522316,"kb_used_data":216192,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93849524,"statfs":{"total":96636764160,"available":96101912576,"internally_reserved":0,"allocated":221380608,"data_stored":221132423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.29099999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:01:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":1.7430000000000001,"15min":0.92300000000000004},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.16500000000000001},"max":{"1min":13.773,"5min":13.773,"15min":13.773},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.5289999999999999,"15min":0.75600000000000001},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":11.957000000000001,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220782592,"data_stored":220779113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161628160,"data_stored":161575382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225271808,"data_stored":225264257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:36.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:37.228 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:37.237 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:37.263 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-09-18T12:02:35.229852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":208396383,"num_objects":50898,"num_object_clones":0,"num_object_copies":101796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244321,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11347,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160929,"ondisk_log_size":160929,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1501424,"kb_used_data":578756,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281614096,"statfs":{"total":289910292480,"available":288372834304,"internally_reserved":0,"allocated":592646144,"data_stored":591851840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,4,4,10,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26935296,"num_objects":-6576,"num_object_clones":0,"num_object_copies":-13152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6576,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002574"},"pg_stats":[{"pgid":"2.f","version":"43'16362","reported_seq":16437,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.664426+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:33.664426+0000","last_peered":"2024-09-18T12:02:33.664426+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:33.664426+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13000704,"num_objects":3174,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16362,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-425}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16274","reported_seq":16436,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:33.700878+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:33.700878+0000","last_peered":"2024-09-18T12:02:33.700878+0000","last_clean":"2024-09-18T12:02:33.700878+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:33.700878+0000","last_undegraded":"2024-09-18T12:02:33.700878+0000","last_fullsized":"2024-09-18T12:02:33.700878+0000","mapping_epoch":29,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12935168,"num_objects":3158,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13466,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16291","reported_seq":16481,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.692266+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:33.692266+0000","last_peered":"2024-09-18T12:02:33.692266+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:33.692266+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13377560,"num_objects":3267,"num_object_clones":0,"num_object_copies":6534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3267,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13149,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-269}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16111","reported_seq":16144,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:33.644979+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:33.644979+0000","last_peered":"2024-09-18T12:02:33.644979+0000","last_clean":"2024-09-18T12:02:33.644979+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:33.644979+0000","last_undegraded":"2024-09-18T12:02:33.644979+0000","last_fullsized":"2024-09-18T12:02:33.644979+0000","mapping_epoch":29,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12898304,"num_objects":3149,"num_object_clones":0,"num_object_copies":6298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16111,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16263","reported_seq":16457,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:33.774948+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:33.774948+0000","last_peered":"2024-09-18T12:02:33.774948+0000","last_clean":"2024-09-18T12:02:33.774948+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:33.774948+0000","last_undegraded":"2024-09-18T12:02:33.774948+0000","last_fullsized":"2024-09-18T12:02:33.774948+0000","mapping_epoch":29,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13041664,"num_objects":3184,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13428,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16180","reported_seq":16265,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.743188+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:33.743188+0000","last_peered":"2024-09-18T12:02:33.743188+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:33.743188+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13284562,"num_objects":3245,"num_object_clones":0,"num_object_copies":6490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16178,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16251","reported_seq":16342,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.752292+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:33.752292+0000","last_peered":"2024-09-18T12:02:33.752292+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:33.752292+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12784850,"num_objects":3123,"num_object_clones":0,"num_object_copies":6246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16249,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16436","reported_seq":16665,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:33.767226+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:33.767226+0000","last_peered":"2024-09-18T12:02:33.767226+0000","last_clean":"2024-09-18T12:02:33.767226+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:33.767226+0000","last_undegraded":"2024-09-18T12:02:33.767226+0000","last_fullsized":"2024-09-18T12:02:33.767226+0000","mapping_epoch":29,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12967936,"num_objects":3166,"num_object_clones":0,"num_object_copies":6332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12753,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16236","reported_seq":23368,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:35.054821+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:35.054821+0000","last_peered":"2024-09-18T12:02:35.054821+0000","last_clean":"2024-09-18T12:02:35.054821+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:35.054821+0000","last_undegraded":"2024-09-18T12:02:35.054821+0000","last_fullsized":"2024-09-18T12:02:35.054821+0000","mapping_epoch":38,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12981458,"num_objects":3171,"num_object_clones":0,"num_object_copies":6342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16234,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16289","reported_seq":17816,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.744383+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:33.744383+0000","last_peered":"2024-09-18T12:02:33.744383+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:33.744383+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12932306,"num_objects":3159,"num_object_clones":0,"num_object_copies":6318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16287,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":724,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":347}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16300","reported_seq":16360,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.663597+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:33.663597+0000","last_peered":"2024-09-18T12:02:33.663597+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:33.663597+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12956882,"num_objects":3165,"num_object_clones":0,"num_object_copies":6330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16298,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16288","reported_seq":23701,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:33.721356+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:33.721356+0000","last_peered":"2024-09-18T12:02:33.721356+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:33.721356+0000","last_undegraded":"2024-09-18T12:02:33.721356+0000","last_fullsized":"2024-09-18T12:02:33.721356+0000","mapping_epoch":42,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13148160,"num_objects":3210,"num_object_clones":0,"num_object_copies":6420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16211","reported_seq":16297,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.725710+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:33.725710+0000","last_peered":"2024-09-18T12:02:33.725710+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:33.725710+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13587049,"num_objects":3318,"num_object_clones":0,"num_object_copies":6636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16210,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16407","reported_seq":23557,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:35.122200+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:35.122200+0000","last_peered":"2024-09-18T12:02:35.122200+0000","last_clean":"2024-09-18T12:02:35.122200+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:35.122200+0000","last_undegraded":"2024-09-18T12:02:35.122200+0000","last_fullsized":"2024-09-18T12:02:35.122200+0000","mapping_epoch":40,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12760891,"num_objects":3118,"num_object_clones":0,"num_object_copies":6236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16404,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16094","reported_seq":16179,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.716990+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:33.716990+0000","last_peered":"2024-09-18T12:02:33.716990+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:33.716990+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12715218,"num_objects":3106,"num_object_clones":0,"num_object_copies":6212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16092,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16333","reported_seq":16422,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.680360+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:33.680360+0000","last_peered":"2024-09-18T12:02:33.680360+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:33.680360+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13023652,"num_objects":3183,"num_object_clones":0,"num_object_copies":6366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16330,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":208396364,"num_objects":50896,"num_object_clones":0,"num_object_copies":101792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50896,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244319,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11346,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589697024,"data_stored":589633216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160926,"ondisk_log_size":160926,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051688,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526716,"kb_used_data":216364,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93845124,"statfs":{"total":96636764160,"available":96097406976,"internally_reserved":0,"allocated":221556736,"data_stored":221308101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.23799999999999999},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.30499999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.41099999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855451,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":454752,"kb_used_data":152716,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93917088,"statfs":{"total":96636764160,"available":96171098112,"internally_reserved":0,"allocated":156381184,"data_stored":156083700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,0,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":6.9139999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":2.5409999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":6.7060000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519956,"kb_used_data":209676,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93851884,"statfs":{"total":96636764160,"available":96104329216,"internally_reserved":0,"allocated":214708224,"data_stored":214460039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":1.486},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":5.875}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":2.4430000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":214007808,"data_stored":214004329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154894336,"data_stored":154841558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220794880,"data_stored":220787329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:37.265 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:37.993 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:38.010 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:38.033 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-18T12:02:37.230343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":208396383,"num_objects":50898,"num_object_clones":0,"num_object_copies":101796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244321,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11347,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160929,"ondisk_log_size":160929,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1501424,"kb_used_data":578756,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281614096,"statfs":{"total":289910292480,"available":288372834304,"internally_reserved":0,"allocated":592646144,"data_stored":591851840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,4,4,10,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17944576,"num_objects":-4381,"num_object_clones":0,"num_object_copies":-8762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4381,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002643"},"pg_stats":[{"pgid":"2.f","version":"43'16362","reported_seq":16437,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.664426+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:33.664426+0000","last_peered":"2024-09-18T12:02:33.664426+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:33.664426+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13000704,"num_objects":3174,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16362,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-425}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16274","reported_seq":16436,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:33.700878+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:33.700878+0000","last_peered":"2024-09-18T12:02:33.700878+0000","last_clean":"2024-09-18T12:02:33.700878+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:33.700878+0000","last_undegraded":"2024-09-18T12:02:33.700878+0000","last_fullsized":"2024-09-18T12:02:33.700878+0000","mapping_epoch":29,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12935168,"num_objects":3158,"num_object_clones":0,"num_object_copies":6316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13466,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16291","reported_seq":16481,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.692266+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:33.692266+0000","last_peered":"2024-09-18T12:02:33.692266+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:33.692266+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13377560,"num_objects":3267,"num_object_clones":0,"num_object_copies":6534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3267,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13149,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-269}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16111","reported_seq":16144,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:33.644979+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:33.644979+0000","last_peered":"2024-09-18T12:02:33.644979+0000","last_clean":"2024-09-18T12:02:33.644979+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:33.644979+0000","last_undegraded":"2024-09-18T12:02:33.644979+0000","last_fullsized":"2024-09-18T12:02:33.644979+0000","mapping_epoch":29,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12898304,"num_objects":3149,"num_object_clones":0,"num_object_copies":6298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16111,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16263","reported_seq":16457,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:33.774948+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:33.774948+0000","last_peered":"2024-09-18T12:02:33.774948+0000","last_clean":"2024-09-18T12:02:33.774948+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:33.774948+0000","last_undegraded":"2024-09-18T12:02:33.774948+0000","last_fullsized":"2024-09-18T12:02:33.774948+0000","mapping_epoch":29,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13041664,"num_objects":3184,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13428,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16180","reported_seq":16265,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.743188+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:33.743188+0000","last_peered":"2024-09-18T12:02:33.743188+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:33.743188+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6100","ondisk_log_start":"37'6100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13284562,"num_objects":3245,"num_object_clones":0,"num_object_copies":6490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16178,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16251","reported_seq":16342,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.752292+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:33.752292+0000","last_peered":"2024-09-18T12:02:33.752292+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:33.752292+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12784850,"num_objects":3123,"num_object_clones":0,"num_object_copies":6246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16249,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-436}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16436","reported_seq":16665,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:33.767226+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:33.767226+0000","last_peered":"2024-09-18T12:02:33.767226+0000","last_clean":"2024-09-18T12:02:33.767226+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:33.767226+0000","last_undegraded":"2024-09-18T12:02:33.767226+0000","last_fullsized":"2024-09-18T12:02:33.767226+0000","mapping_epoch":29,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12967936,"num_objects":3166,"num_object_clones":0,"num_object_copies":6332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12753,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16236","reported_seq":23368,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:35.054821+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:35.054821+0000","last_peered":"2024-09-18T12:02:35.054821+0000","last_clean":"2024-09-18T12:02:35.054821+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:35.054821+0000","last_undegraded":"2024-09-18T12:02:35.054821+0000","last_fullsized":"2024-09-18T12:02:35.054821+0000","mapping_epoch":38,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12981458,"num_objects":3171,"num_object_clones":0,"num_object_copies":6342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16234,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16289","reported_seq":17816,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.744383+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:33.744383+0000","last_peered":"2024-09-18T12:02:33.744383+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:33.744383+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12932306,"num_objects":3159,"num_object_clones":0,"num_object_copies":6318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16287,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":724,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":347}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16300","reported_seq":16360,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.663597+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:33.663597+0000","last_peered":"2024-09-18T12:02:33.663597+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:33.663597+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12956882,"num_objects":3165,"num_object_clones":0,"num_object_copies":6330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16298,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16288","reported_seq":23701,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:33.721356+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:33.721356+0000","last_peered":"2024-09-18T12:02:33.721356+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:33.721356+0000","last_undegraded":"2024-09-18T12:02:33.721356+0000","last_fullsized":"2024-09-18T12:02:33.721356+0000","mapping_epoch":42,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13148160,"num_objects":3210,"num_object_clones":0,"num_object_copies":6420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3210}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16211","reported_seq":16297,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.725710+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:33.725710+0000","last_peered":"2024-09-18T12:02:33.725710+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:33.725710+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13587049,"num_objects":3318,"num_object_clones":0,"num_object_copies":6636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16210,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16407","reported_seq":23557,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:35.122200+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:35.122200+0000","last_peered":"2024-09-18T12:02:35.122200+0000","last_clean":"2024-09-18T12:02:35.122200+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:35.122200+0000","last_undegraded":"2024-09-18T12:02:35.122200+0000","last_fullsized":"2024-09-18T12:02:35.122200+0000","mapping_epoch":40,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12760891,"num_objects":3118,"num_object_clones":0,"num_object_copies":6236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16404,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16094","reported_seq":16179,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.716990+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:33.716990+0000","last_peered":"2024-09-18T12:02:33.716990+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:33.716990+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6000","ondisk_log_start":"37'6000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12715218,"num_objects":3106,"num_object_clones":0,"num_object_copies":6212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16092,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-487}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16333","reported_seq":16422,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:33.680360+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:33.680360+0000","last_peered":"2024-09-18T12:02:33.680360+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:33.680360+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13023652,"num_objects":3183,"num_object_clones":0,"num_object_copies":6366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16330,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":208396364,"num_objects":50896,"num_object_clones":0,"num_object_copies":101792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50896,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244319,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11346,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589697024,"data_stored":589633216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160926,"ondisk_log_size":160926,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051688,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526716,"kb_used_data":216364,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93845124,"statfs":{"total":96636764160,"available":96097406976,"internally_reserved":0,"allocated":221556736,"data_stored":221308101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.23799999999999999},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.30499999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.41099999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855451,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":454752,"kb_used_data":152716,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93917088,"statfs":{"total":96636764160,"available":96171098112,"internally_reserved":0,"allocated":156381184,"data_stored":156083700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,0,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":6.9139999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":2.5409999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":6.7060000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519956,"kb_used_data":209676,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93851884,"statfs":{"total":96636764160,"available":96104329216,"internally_reserved":0,"allocated":214708224,"data_stored":214460039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":1.486},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":5.875}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":2.4430000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":214007808,"data_stored":214004329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154894336,"data_stored":154841558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220794880,"data_stored":220787329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:38.034 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:02:38.034 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'16362", 'reported_seq': 16437, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:33.664426+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:02:33.664426+0000', 'last_peered': '2024-09-18T12:02:33.664426+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:02:33.664426+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'6300", 'ondisk_log_start': "37'6300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13000704, 'num_objects': 3174, 'num_object_clones': 0, 'num_object_copies': 6348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16362, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -425}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:38.034 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:02:38.034 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'16291", 'reported_seq': 16481, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:33.692266+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:02:33.692266+0000', 'last_peered': '2024-09-18T12:02:33.692266+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:02:33.692266+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'6200", 'ondisk_log_start': "37'6200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13377560, 'num_objects': 3267, 'num_object_clones': 0, 'num_object_copies': 6534, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3267, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13149, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -269}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:38.034 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:02:38.034 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'16180", 'reported_seq': 16265, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:33.743188+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:02:33.743188+0000', 'last_peered': '2024-09-18T12:02:33.743188+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:02:33.743188+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'6100", 'ondisk_log_start': "37'6100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13284562, 'num_objects': 3245, 'num_object_clones': 0, 'num_object_copies': 6490, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16178, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -282}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:38.035 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:02:38.035 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'16251", 'reported_seq': 16342, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:33.752292+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:02:33.752292+0000', 'last_peered': '2024-09-18T12:02:33.752292+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:02:33.752292+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'6200", 'ondisk_log_start': "37'6200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12784850, 'num_objects': 3123, 'num_object_clones': 0, 'num_object_copies': 6246, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16249, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -436}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:38.035 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:02:38.035 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'16289", 'reported_seq': 17816, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:33.744383+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:02:33.744383+0000', 'last_peered': '2024-09-18T12:02:33.744383+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:02:33.744383+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'6200", 'ondisk_log_start': "37'6200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12932306, 'num_objects': 3159, 'num_object_clones': 0, 'num_object_copies': 6318, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2812, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16287, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 724, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 347}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:38.035 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:02:38.035 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'16300", 'reported_seq': 16360, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:33.663597+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:02:33.663597+0000', 'last_peered': '2024-09-18T12:02:33.663597+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:02:33.663597+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'6200", 'ondisk_log_start': "37'6200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12956882, 'num_objects': 3165, 'num_object_clones': 0, 'num_object_copies': 6330, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16298, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -385}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:38.035 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:02:38.035 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'16288", 'reported_seq': 23701, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:02:33.721356+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:02:33.721356+0000', 'last_peered': '2024-09-18T12:02:33.721356+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:02:33.721356+0000', 'last_undegraded': '2024-09-18T12:02:33.721356+0000', 'last_fullsized': '2024-09-18T12:02:33.721356+0000', 'mapping_epoch': 42, 'log_start': "37'6200", 'ondisk_log_start': "37'6200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13148160, 'num_objects': 3210, 'num_object_clones': 0, 'num_object_copies': 6420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12768, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 3210}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:38.035 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:02:38.035 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'16211", 'reported_seq': 16297, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:33.725710+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:02:33.725710+0000', 'last_peered': '2024-09-18T12:02:33.725710+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:02:33.725710+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'6200", 'ondisk_log_start': "37'6200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13587049, 'num_objects': 3318, 'num_object_clones': 0, 'num_object_copies': 6636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16210, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:38.036 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:02:38.036 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'16094", 'reported_seq': 16179, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:33.716990+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:02:33.716990+0000', 'last_peered': '2024-09-18T12:02:33.716990+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:02:33.716990+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'6000", 'ondisk_log_start': "37'6000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12715218, 'num_objects': 3106, 'num_object_clones': 0, 'num_object_copies': 6212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16092, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -487}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:38.036 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:02:38.036 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'16333", 'reported_seq': 16422, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:33.680360+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:02:33.680360+0000', 'last_peered': '2024-09-18T12:02:33.680360+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:02:33.680360+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'6300", 'ondisk_log_start': "37'6300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13023652, 'num_objects': 3183, 'num_object_clones': 0, 'num_object_copies': 6366, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16330, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:38.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:02:38.917 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:38.917 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31247/101796 objects degraded (30.696%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":558,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":50898,"data_bytes":208396383,"bytes_used":1537458176,"bytes_avail":288372834304,"bytes_total":289910292480,"degraded_objects":31247,"degraded_total":101796,"degraded_ratio":0.30695705135761719,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":365},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:02:38.966 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:02:38.966 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:39.617 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:39.648 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:39.669 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-18T12:02:39.230788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":200581215,"num_objects":48990,"num_object_clones":0,"num_object_copies":97980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48990,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246229,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11418,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160737,"ondisk_log_size":160737,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1469960,"kb_used_data":568100,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281645560,"statfs":{"total":289910292480,"available":288405053440,"internally_reserved":0,"allocated":581734400,"data_stored":580940096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,3,18,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25759744,"num_objects":-6289,"num_object_clones":0,"num_object_copies":-12578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6289,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002615"},"pg_stats":[{"pgid":"2.f","version":"43'16504","reported_seq":16579,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.647829+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:38.647829+0000","last_peered":"2024-09-18T12:02:38.647829+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:38.647829+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12419072,"num_objects":3032,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16504,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-567}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16400","reported_seq":16562,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:38.733749+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:38.733749+0000","last_peered":"2024-09-18T12:02:38.733749+0000","last_clean":"2024-09-18T12:02:38.733749+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:38.733749+0000","last_undegraded":"2024-09-18T12:02:38.733749+0000","last_fullsized":"2024-09-18T12:02:38.733749+0000","mapping_epoch":29,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12419072,"num_objects":3032,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13592,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16439","reported_seq":16628,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.750979+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:38.750979+0000","last_peered":"2024-09-18T12:02:38.750979+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:38.750979+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12771352,"num_objects":3119,"num_object_clones":0,"num_object_copies":6238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13297,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-417}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16232","reported_seq":16265,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:38.712967+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:38.712967+0000","last_peered":"2024-09-18T12:02:38.712967+0000","last_clean":"2024-09-18T12:02:38.712967+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:38.712967+0000","last_undegraded":"2024-09-18T12:02:38.712967+0000","last_fullsized":"2024-09-18T12:02:38.712967+0000","mapping_epoch":29,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12402688,"num_objects":3028,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16393","reported_seq":16588,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:38.794646+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:38.794646+0000","last_peered":"2024-09-18T12:02:38.794646+0000","last_clean":"2024-09-18T12:02:38.794646+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:38.794646+0000","last_undegraded":"2024-09-18T12:02:38.794646+0000","last_fullsized":"2024-09-18T12:02:38.794646+0000","mapping_epoch":29,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12509184,"num_objects":3054,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13558,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16329","reported_seq":16414,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.784491+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:38.784491+0000","last_peered":"2024-09-18T12:02:38.784491+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:38.784491+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12674258,"num_objects":3096,"num_object_clones":0,"num_object_copies":6192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16327,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16382","reported_seq":16473,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.813777+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:38.813777+0000","last_peered":"2024-09-18T12:02:38.813777+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:38.813777+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12248274,"num_objects":2992,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16380,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16555","reported_seq":16784,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:38.757966+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:38.757966+0000","last_peered":"2024-09-18T12:02:38.757966+0000","last_clean":"2024-09-18T12:02:38.757966+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:38.757966+0000","last_undegraded":"2024-09-18T12:02:38.757966+0000","last_fullsized":"2024-09-18T12:02:38.757966+0000","mapping_epoch":29,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12480512,"num_objects":3047,"num_object_clones":0,"num_object_copies":6094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12872,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16236","reported_seq":23368,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:35.054821+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:35.054821+0000","last_peered":"2024-09-18T12:02:35.054821+0000","last_clean":"2024-09-18T12:02:35.054821+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:35.054821+0000","last_undegraded":"2024-09-18T12:02:35.054821+0000","last_fullsized":"2024-09-18T12:02:35.054821+0000","mapping_epoch":38,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12981458,"num_objects":3171,"num_object_clones":0,"num_object_copies":6342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16234,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16429","reported_seq":18098,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.743274+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:38.743274+0000","last_peered":"2024-09-18T12:02:38.743274+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:38.743274+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12358866,"num_objects":3019,"num_object_clones":0,"num_object_copies":6038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16427,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":795,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":278}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16434","reported_seq":16494,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.703418+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:38.703418+0000","last_peered":"2024-09-18T12:02:38.703418+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:38.703418+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12408018,"num_objects":3031,"num_object_clones":0,"num_object_copies":6062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16432,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-519}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16411","reported_seq":23824,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:38.682023+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:38.682023+0000","last_peered":"2024-09-18T12:02:38.682023+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:38.682023+0000","last_undegraded":"2024-09-18T12:02:38.682023+0000","last_fullsized":"2024-09-18T12:02:38.682023+0000","mapping_epoch":42,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12644352,"num_objects":3087,"num_object_clones":0,"num_object_copies":6174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3087}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16361","reported_seq":16447,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.766621+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:38.766621+0000","last_peered":"2024-09-18T12:02:38.766621+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:38.766621+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12972649,"num_objects":3168,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16360,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16407","reported_seq":23557,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:35.122200+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:35.122200+0000","last_peered":"2024-09-18T12:02:35.122200+0000","last_clean":"2024-09-18T12:02:35.122200+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:35.122200+0000","last_undegraded":"2024-09-18T12:02:35.122200+0000","last_fullsized":"2024-09-18T12:02:35.122200+0000","mapping_epoch":40,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12760891,"num_objects":3118,"num_object_clones":0,"num_object_copies":6236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16404,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16244","reported_seq":16325,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.801287+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:38.801287+0000","last_peered":"2024-09-18T12:02:38.801287+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:38.801287+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12100818,"num_objects":2956,"num_object_clones":0,"num_object_copies":5912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16242,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16478","reported_seq":16567,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.779365+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:38.779365+0000","last_peered":"2024-09-18T12:02:38.779365+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:38.779365+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12429732,"num_objects":3038,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16475,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":200581196,"num_objects":48988,"num_object_clones":0,"num_object_copies":97976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48988,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246227,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11417,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":578719744,"data_stored":578655936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160734,"ondisk_log_size":160734,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051689,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509992,"kb_used_data":212128,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93861848,"statfs":{"total":96636764160,"available":96114532352,"internally_reserved":0,"allocated":217219072,"data_stored":216970437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.19600000000000001},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.33800000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.24099999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.27000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855451,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":454752,"kb_used_data":152716,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93917088,"statfs":{"total":96636764160,"available":96171098112,"internally_reserved":0,"allocated":156381184,"data_stored":156083700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,0,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":6.9139999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":2.5409999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":6.7060000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505216,"kb_used_data":203256,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93866624,"statfs":{"total":96636764160,"available":96119422976,"internally_reserved":0,"allocated":208134144,"data_stored":207885959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,9,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":4.5259999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.224}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":11.071999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":4.468}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":207347712,"data_stored":207344233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154894336,"data_stored":154841558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216477696,"data_stored":216470145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:42.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:43.424 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:43.434 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:43.446 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-09-18T12:02:43.231948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":199516255,"num_objects":48730,"num_object_clones":0,"num_object_copies":97460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48730,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246489,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11418,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160797,"ondisk_log_size":160797,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1471740,"kb_used_data":561572,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281643780,"statfs":{"total":289910292480,"available":288403230720,"internally_reserved":0,"allocated":575049728,"data_stored":574255424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,15,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17756160,"num_objects":-4335,"num_object_clones":0,"num_object_copies":-8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"2.f","version":"43'16504","reported_seq":16579,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.647829+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:38.647829+0000","last_peered":"2024-09-18T12:02:38.647829+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:38.647829+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12419072,"num_objects":3032,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16504,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-567}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16400","reported_seq":16562,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:38.733749+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:38.733749+0000","last_peered":"2024-09-18T12:02:38.733749+0000","last_clean":"2024-09-18T12:02:38.733749+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:38.733749+0000","last_undegraded":"2024-09-18T12:02:38.733749+0000","last_fullsized":"2024-09-18T12:02:38.733749+0000","mapping_epoch":29,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12419072,"num_objects":3032,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13592,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16439","reported_seq":16628,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.750979+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:38.750979+0000","last_peered":"2024-09-18T12:02:38.750979+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:38.750979+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12771352,"num_objects":3119,"num_object_clones":0,"num_object_copies":6238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13297,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-417}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16232","reported_seq":16265,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:38.712967+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:38.712967+0000","last_peered":"2024-09-18T12:02:38.712967+0000","last_clean":"2024-09-18T12:02:38.712967+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:38.712967+0000","last_undegraded":"2024-09-18T12:02:38.712967+0000","last_fullsized":"2024-09-18T12:02:38.712967+0000","mapping_epoch":29,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12402688,"num_objects":3028,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16393","reported_seq":16588,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:38.794646+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:38.794646+0000","last_peered":"2024-09-18T12:02:38.794646+0000","last_clean":"2024-09-18T12:02:38.794646+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:38.794646+0000","last_undegraded":"2024-09-18T12:02:38.794646+0000","last_fullsized":"2024-09-18T12:02:38.794646+0000","mapping_epoch":29,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12509184,"num_objects":3054,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13558,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16329","reported_seq":16414,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.784491+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:38.784491+0000","last_peered":"2024-09-18T12:02:38.784491+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:38.784491+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12674258,"num_objects":3096,"num_object_clones":0,"num_object_copies":6192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16327,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16382","reported_seq":16473,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.813777+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:38.813777+0000","last_peered":"2024-09-18T12:02:38.813777+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:38.813777+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12248274,"num_objects":2992,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16380,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16555","reported_seq":16784,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:38.757966+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:38.757966+0000","last_peered":"2024-09-18T12:02:38.757966+0000","last_clean":"2024-09-18T12:02:38.757966+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:38.757966+0000","last_undegraded":"2024-09-18T12:02:38.757966+0000","last_fullsized":"2024-09-18T12:02:38.757966+0000","mapping_epoch":29,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12480512,"num_objects":3047,"num_object_clones":0,"num_object_copies":6094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12872,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16367","reported_seq":23499,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:40.210074+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:40.210074+0000","last_peered":"2024-09-18T12:02:40.210074+0000","last_clean":"2024-09-18T12:02:40.210074+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:40.210074+0000","last_undegraded":"2024-09-18T12:02:40.210074+0000","last_fullsized":"2024-09-18T12:02:40.210074+0000","mapping_epoch":38,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12444882,"num_objects":3040,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16365,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16429","reported_seq":18098,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.743274+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:38.743274+0000","last_peered":"2024-09-18T12:02:38.743274+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:38.743274+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12358866,"num_objects":3019,"num_object_clones":0,"num_object_copies":6038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16427,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":795,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":278}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16434","reported_seq":16494,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.703418+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:38.703418+0000","last_peered":"2024-09-18T12:02:38.703418+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:38.703418+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12408018,"num_objects":3031,"num_object_clones":0,"num_object_copies":6062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16432,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-519}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16411","reported_seq":23824,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:38.682023+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:38.682023+0000","last_peered":"2024-09-18T12:02:38.682023+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:38.682023+0000","last_undegraded":"2024-09-18T12:02:38.682023+0000","last_fullsized":"2024-09-18T12:02:38.682023+0000","mapping_epoch":42,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12644352,"num_objects":3087,"num_object_clones":0,"num_object_copies":6174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3087}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16361","reported_seq":16447,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.766621+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:38.766621+0000","last_peered":"2024-09-18T12:02:38.766621+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:38.766621+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12972649,"num_objects":3168,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16360,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16536","reported_seq":23686,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:40.238699+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:40.238699+0000","last_peered":"2024-09-18T12:02:40.238699+0000","last_clean":"2024-09-18T12:02:40.238699+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:40.238699+0000","last_undegraded":"2024-09-18T12:02:40.238699+0000","last_fullsized":"2024-09-18T12:02:40.238699+0000","mapping_epoch":40,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12232507,"num_objects":2989,"num_object_clones":0,"num_object_copies":5978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16533,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16244","reported_seq":16325,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.801287+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:38.801287+0000","last_peered":"2024-09-18T12:02:38.801287+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:38.801287+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12100818,"num_objects":2956,"num_object_clones":0,"num_object_copies":5912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16242,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16478","reported_seq":16567,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.779365+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:38.779365+0000","last_peered":"2024-09-18T12:02:38.779365+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:38.779365+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12429732,"num_objects":3038,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16475,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":199516236,"num_objects":48728,"num_object_clones":0,"num_object_copies":97456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48728,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246487,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11417,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":571879424,"data_stored":571815616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160794,"ondisk_log_size":160794,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051689,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509992,"kb_used_data":212128,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93861848,"statfs":{"total":96636764160,"available":96114532352,"internally_reserved":0,"allocated":217219072,"data_stored":216970437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.19600000000000001},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.33800000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.24099999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.27000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855452,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456532,"kb_used_data":146188,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93915308,"statfs":{"total":96636764160,"available":96169275392,"internally_reserved":0,"allocated":149696512,"data_stored":149399028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,0,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":2.879}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":5.3620000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.32200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505216,"kb_used_data":203256,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93866624,"statfs":{"total":96636764160,"available":96119422976,"internally_reserved":0,"allocated":208134144,"data_stored":207885959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,9,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":4.5259999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.224}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":11.071999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":4.468}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":207347712,"data_stored":207344233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148054016,"data_stored":148001238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216477696,"data_stored":216470145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:43.448 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:44.090 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:44.108 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:44.122 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-09-18T12:02:43.231948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":199516255,"num_objects":48730,"num_object_clones":0,"num_object_copies":97460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48730,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246489,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11418,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160797,"ondisk_log_size":160797,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1471740,"kb_used_data":561572,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281643780,"statfs":{"total":289910292480,"available":288403230720,"internally_reserved":0,"allocated":575049728,"data_stored":574255424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,15,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17756160,"num_objects":-4335,"num_object_clones":0,"num_object_copies":-8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"2.f","version":"43'16504","reported_seq":16579,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.647829+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:38.647829+0000","last_peered":"2024-09-18T12:02:38.647829+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:38.647829+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12419072,"num_objects":3032,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16504,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-567}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16400","reported_seq":16562,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:38.733749+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:38.733749+0000","last_peered":"2024-09-18T12:02:38.733749+0000","last_clean":"2024-09-18T12:02:38.733749+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:38.733749+0000","last_undegraded":"2024-09-18T12:02:38.733749+0000","last_fullsized":"2024-09-18T12:02:38.733749+0000","mapping_epoch":29,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12419072,"num_objects":3032,"num_object_clones":0,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13592,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16439","reported_seq":16628,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.750979+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:38.750979+0000","last_peered":"2024-09-18T12:02:38.750979+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:38.750979+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12771352,"num_objects":3119,"num_object_clones":0,"num_object_copies":6238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13297,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-417}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16232","reported_seq":16265,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:38.712967+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:38.712967+0000","last_peered":"2024-09-18T12:02:38.712967+0000","last_clean":"2024-09-18T12:02:38.712967+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:38.712967+0000","last_undegraded":"2024-09-18T12:02:38.712967+0000","last_fullsized":"2024-09-18T12:02:38.712967+0000","mapping_epoch":29,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12402688,"num_objects":3028,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16232,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16393","reported_seq":16588,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:38.794646+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:38.794646+0000","last_peered":"2024-09-18T12:02:38.794646+0000","last_clean":"2024-09-18T12:02:38.794646+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:38.794646+0000","last_undegraded":"2024-09-18T12:02:38.794646+0000","last_fullsized":"2024-09-18T12:02:38.794646+0000","mapping_epoch":29,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12509184,"num_objects":3054,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13558,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16329","reported_seq":16414,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.784491+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:38.784491+0000","last_peered":"2024-09-18T12:02:38.784491+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:38.784491+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12674258,"num_objects":3096,"num_object_clones":0,"num_object_copies":6192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16327,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16382","reported_seq":16473,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.813777+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:38.813777+0000","last_peered":"2024-09-18T12:02:38.813777+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:38.813777+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12248274,"num_objects":2992,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16380,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16555","reported_seq":16784,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:38.757966+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:38.757966+0000","last_peered":"2024-09-18T12:02:38.757966+0000","last_clean":"2024-09-18T12:02:38.757966+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:38.757966+0000","last_undegraded":"2024-09-18T12:02:38.757966+0000","last_fullsized":"2024-09-18T12:02:38.757966+0000","mapping_epoch":29,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12480512,"num_objects":3047,"num_object_clones":0,"num_object_copies":6094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12872,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16367","reported_seq":23499,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:40.210074+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:40.210074+0000","last_peered":"2024-09-18T12:02:40.210074+0000","last_clean":"2024-09-18T12:02:40.210074+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:40.210074+0000","last_undegraded":"2024-09-18T12:02:40.210074+0000","last_fullsized":"2024-09-18T12:02:40.210074+0000","mapping_epoch":38,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12444882,"num_objects":3040,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16365,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16429","reported_seq":18098,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.743274+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:38.743274+0000","last_peered":"2024-09-18T12:02:38.743274+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:38.743274+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12358866,"num_objects":3019,"num_object_clones":0,"num_object_copies":6038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16427,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":795,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":278}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16434","reported_seq":16494,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.703418+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:38.703418+0000","last_peered":"2024-09-18T12:02:38.703418+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:38.703418+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12408018,"num_objects":3031,"num_object_clones":0,"num_object_copies":6062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16432,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-519}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16411","reported_seq":23824,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:38.682023+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:38.682023+0000","last_peered":"2024-09-18T12:02:38.682023+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:38.682023+0000","last_undegraded":"2024-09-18T12:02:38.682023+0000","last_fullsized":"2024-09-18T12:02:38.682023+0000","mapping_epoch":42,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12644352,"num_objects":3087,"num_object_clones":0,"num_object_copies":6174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":3087}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16361","reported_seq":16447,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.766621+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:38.766621+0000","last_peered":"2024-09-18T12:02:38.766621+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:38.766621+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12972649,"num_objects":3168,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16360,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16536","reported_seq":23686,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:40.238699+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:40.238699+0000","last_peered":"2024-09-18T12:02:40.238699+0000","last_clean":"2024-09-18T12:02:40.238699+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:40.238699+0000","last_undegraded":"2024-09-18T12:02:40.238699+0000","last_fullsized":"2024-09-18T12:02:40.238699+0000","mapping_epoch":40,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12232507,"num_objects":2989,"num_object_clones":0,"num_object_copies":5978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16533,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16244","reported_seq":16325,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.801287+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:38.801287+0000","last_peered":"2024-09-18T12:02:38.801287+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:38.801287+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6200","ondisk_log_start":"37'6200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12100818,"num_objects":2956,"num_object_clones":0,"num_object_copies":5912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16242,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16478","reported_seq":16567,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:38.779365+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:38.779365+0000","last_peered":"2024-09-18T12:02:38.779365+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:38.779365+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12429732,"num_objects":3038,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16475,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-485}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":199516236,"num_objects":48728,"num_object_clones":0,"num_object_copies":97456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48728,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246487,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11417,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":571879424,"data_stored":571815616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160794,"ondisk_log_size":160794,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051689,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509992,"kb_used_data":212128,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93861848,"statfs":{"total":96636764160,"available":96114532352,"internally_reserved":0,"allocated":217219072,"data_stored":216970437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.19600000000000001},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.33800000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.24099999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.27000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855452,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456532,"kb_used_data":146188,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93915308,"statfs":{"total":96636764160,"available":96169275392,"internally_reserved":0,"allocated":149696512,"data_stored":149399028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,0,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":2.879}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":5.3620000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.32200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505216,"kb_used_data":203256,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93866624,"statfs":{"total":96636764160,"available":96119422976,"internally_reserved":0,"allocated":208134144,"data_stored":207885959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,9,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":4.5259999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.224}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":11.071999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":4.468}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":207347712,"data_stored":207344233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148054016,"data_stored":148001238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216477696,"data_stored":216470145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:44.124 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:02:44.124 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'16504", 'reported_seq': 16579, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:38.647829+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:02:38.647829+0000', 'last_peered': '2024-09-18T12:02:38.647829+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:02:38.647829+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'6500", 'ondisk_log_start': "37'6500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12419072, 'num_objects': 3032, 'num_object_clones': 0, 'num_object_copies': 6064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16504, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -567}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:44.124 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:02:44.124 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'16439", 'reported_seq': 16628, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:38.750979+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:02:38.750979+0000', 'last_peered': '2024-09-18T12:02:38.750979+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:02:38.750979+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'6400", 'ondisk_log_start': "37'6400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12771352, 'num_objects': 3119, 'num_object_clones': 0, 'num_object_copies': 6238, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3119, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13297, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -417}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:44.124 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:02:44.124 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'16329", 'reported_seq': 16414, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:38.784491+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:02:38.784491+0000', 'last_peered': '2024-09-18T12:02:38.784491+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:02:38.784491+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'6300", 'ondisk_log_start': "37'6300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12674258, 'num_objects': 3096, 'num_object_clones': 0, 'num_object_copies': 6192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16327, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -431}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:44.124 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:02:44.124 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'16382", 'reported_seq': 16473, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:38.813777+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:02:38.813777+0000', 'last_peered': '2024-09-18T12:02:38.813777+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:02:38.813777+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'6300", 'ondisk_log_start': "37'6300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12248274, 'num_objects': 2992, 'num_object_clones': 0, 'num_object_copies': 5984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16380, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -567}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:44.125 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:02:44.125 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'16429", 'reported_seq': 18098, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:38.743274+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:02:38.743274+0000', 'last_peered': '2024-09-18T12:02:38.743274+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:02:38.743274+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'6400", 'ondisk_log_start': "37'6400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12358866, 'num_objects': 3019, 'num_object_clones': 0, 'num_object_copies': 6038, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2741, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16427, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 795, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 278}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:44.125 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:02:44.125 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'16434", 'reported_seq': 16494, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:38.703418+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:02:38.703418+0000', 'last_peered': '2024-09-18T12:02:38.703418+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:02:38.703418+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'6400", 'ondisk_log_start': "37'6400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12408018, 'num_objects': 3031, 'num_object_clones': 0, 'num_object_copies': 6062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3031, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16432, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -519}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:44.125 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:02:44.125 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'16411", 'reported_seq': 23824, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:02:38.682023+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:02:38.682023+0000', 'last_peered': '2024-09-18T12:02:38.682023+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:02:38.682023+0000', 'last_undegraded': '2024-09-18T12:02:38.682023+0000', 'last_fullsized': '2024-09-18T12:02:38.682023+0000', 'mapping_epoch': 42, 'log_start': "37'6400", 'ondisk_log_start': "37'6400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12644352, 'num_objects': 3087, 'num_object_clones': 0, 'num_object_copies': 6174, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 3087}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:44.125 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:02:44.125 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'16361", 'reported_seq': 16447, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:38.766621+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:02:38.766621+0000', 'last_peered': '2024-09-18T12:02:38.766621+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:02:38.766621+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'6300", 'ondisk_log_start': "37'6300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12972649, 'num_objects': 3168, 'num_object_clones': 0, 'num_object_copies': 6336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16360, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -367}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:44.125 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:02:44.125 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'16244", 'reported_seq': 16325, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:38.801287+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:02:38.801287+0000', 'last_peered': '2024-09-18T12:02:38.801287+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:02:38.801287+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'6200", 'ondisk_log_start': "37'6200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12100818, 'num_objects': 2956, 'num_object_clones': 0, 'num_object_copies': 5912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16242, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -637}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:44.126 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:02:44.126 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'16478", 'reported_seq': 16567, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:38.779365+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:02:38.779365+0000', 'last_peered': '2024-09-18T12:02:38.779365+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:02:38.779365+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'6400", 'ondisk_log_start': "37'6400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12429732, 'num_objects': 3038, 'num_object_clones': 0, 'num_object_copies': 6076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16475, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -485}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:44.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:02:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:45.144 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31176/97460 objects degraded (31.989%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":565,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":48730,"data_bytes":199516255,"bytes_used":1507061760,"bytes_avail":288403230720,"bytes_total":289910292480,"degraded_objects":31176,"degraded_total":97460,"degraded_ratio":0.31988508105889596,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":361},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:02:45.174 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:02:45.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:45.840 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:45.862 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-18T12:02:45.232363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":191594591,"num_objects":46796,"num_object_clones":0,"num_object_copies":93592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46796,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":248423,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11499,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160831,"ondisk_log_size":160831,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1468864,"kb_used_data":550372,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281646656,"statfs":{"total":289910292480,"available":288406175744,"internally_reserved":0,"allocated":563580928,"data_stored":562786624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,7,6,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25677824,"num_objects":-6269,"num_object_clones":0,"num_object_copies":-12538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6269,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002877"},"pg_stats":[{"pgid":"2.f","version":"43'16639","reported_seq":16714,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:43.726038+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:43.726038+0000","last_peered":"2024-09-18T12:02:43.726038+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:43.726038+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11866112,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16639,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-702}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16524","reported_seq":16683,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:43.711641+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:43.711641+0000","last_peered":"2024-09-18T12:02:43.711641+0000","last_clean":"2024-09-18T12:02:43.711641+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:43.711641+0000","last_undegraded":"2024-09-18T12:02:43.711641+0000","last_fullsized":"2024-09-18T12:02:43.711641+0000","mapping_epoch":29,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13716,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16591","reported_seq":16781,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:43.746016+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:43.746016+0000","last_peered":"2024-09-18T12:02:43.746016+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:43.746016+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12148760,"num_objects":2967,"num_object_clones":0,"num_object_copies":5934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13449,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-569}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16367","reported_seq":16400,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:43.677531+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:43.677531+0000","last_peered":"2024-09-18T12:02:43.677531+0000","last_clean":"2024-09-18T12:02:43.677531+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:43.677531+0000","last_undegraded":"2024-09-18T12:02:43.677531+0000","last_fullsized":"2024-09-18T12:02:43.677531+0000","mapping_epoch":29,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11849728,"num_objects":2893,"num_object_clones":0,"num_object_copies":5786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16367,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16550","reported_seq":16745,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:43.685008+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:43.685008+0000","last_peered":"2024-09-18T12:02:43.685008+0000","last_clean":"2024-09-18T12:02:43.685008+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:43.685008+0000","last_undegraded":"2024-09-18T12:02:43.685008+0000","last_fullsized":"2024-09-18T12:02:43.685008+0000","mapping_epoch":29,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11866112,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13715,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16465","reported_seq":16550,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:43.808038+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:43.808038+0000","last_peered":"2024-09-18T12:02:43.808038+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:43.808038+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12117202,"num_objects":2960,"num_object_clones":0,"num_object_copies":5920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16463,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-567}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16514","reported_seq":16605,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:43.780237+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:43.780237+0000","last_peered":"2024-09-18T12:02:43.780237+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:43.780237+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11707602,"num_objects":2860,"num_object_clones":0,"num_object_copies":5720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16512,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16695","reported_seq":16924,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:43.686933+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:43.686933+0000","last_peered":"2024-09-18T12:02:43.686933+0000","last_clean":"2024-09-18T12:02:43.686933+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:43.686933+0000","last_undegraded":"2024-09-18T12:02:43.686933+0000","last_fullsized":"2024-09-18T12:02:43.686933+0000","mapping_epoch":29,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11907072,"num_objects":2907,"num_object_clones":0,"num_object_copies":5814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16367","reported_seq":23499,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:40.210074+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:40.210074+0000","last_peered":"2024-09-18T12:02:40.210074+0000","last_clean":"2024-09-18T12:02:40.210074+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:40.210074+0000","last_undegraded":"2024-09-18T12:02:40.210074+0000","last_fullsized":"2024-09-18T12:02:40.210074+0000","mapping_epoch":38,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12444882,"num_objects":3040,"num_object_clones":0,"num_object_copies":6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16365,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16560","reported_seq":18391,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:43.810569+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:43.810569+0000","last_peered":"2024-09-18T12:02:43.810569+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:43.810569+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11822290,"num_objects":2888,"num_object_clones":0,"num_object_copies":5776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16558,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":876,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":228}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16577","reported_seq":16637,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:43.491613+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:43.491613+0000","last_peered":"2024-09-18T12:02:43.491613+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:43.491613+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11822290,"num_objects":2888,"num_object_clones":0,"num_object_copies":5776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16575,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-662}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16543","reported_seq":23956,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:43.693227+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:43.693227+0000","last_peered":"2024-09-18T12:02:43.693227+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:43.693227+0000","last_undegraded":"2024-09-18T12:02:43.693227+0000","last_fullsized":"2024-09-18T12:02:43.693227+0000","mapping_epoch":42,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12103680,"num_objects":2955,"num_object_clones":0,"num_object_copies":5910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13023,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16514","reported_seq":16599,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:43.801877+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:43.801877+0000","last_peered":"2024-09-18T12:02:43.801877+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:43.801877+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12345961,"num_objects":3015,"num_object_clones":0,"num_object_copies":6030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16513,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16536","reported_seq":23686,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:40.238699+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:40.238699+0000","last_peered":"2024-09-18T12:02:40.238699+0000","last_clean":"2024-09-18T12:02:40.238699+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:40.238699+0000","last_undegraded":"2024-09-18T12:02:40.238699+0000","last_fullsized":"2024-09-18T12:02:40.238699+0000","mapping_epoch":40,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12232507,"num_objects":2989,"num_object_clones":0,"num_object_copies":5978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16533,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16364","reported_seq":16449,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:43.821539+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:43.821539+0000","last_peered":"2024-09-18T12:02:43.821539+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:43.821539+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6300","ondisk_log_start":"37'6300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11609298,"num_objects":2836,"num_object_clones":0,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16362,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-757}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16622","reported_seq":16710,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:43.800062+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:43.800062+0000","last_peered":"2024-09-18T12:02:43.800062+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:43.800062+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11839908,"num_objects":2894,"num_object_clones":0,"num_object_copies":5788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16619,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-629}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":191594572,"num_objects":46794,"num_object_clones":0,"num_object_copies":93588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46794,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":248421,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11498,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":560566272,"data_stored":560502464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160828,"ondisk_log_size":160828,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051690,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509736,"kb_used_data":207708,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93862104,"statfs":{"total":96636764160,"available":96114794496,"internally_reserved":0,"allocated":212692992,"data_stored":212444357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.45500000000000002},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":5.5629999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":5.5110000000000001},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.26600000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855452,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456532,"kb_used_data":146188,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93915308,"statfs":{"total":96636764160,"available":96169275392,"internally_reserved":0,"allocated":149696512,"data_stored":149399028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,0,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":2.879}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":5.3620000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.32200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":502596,"kb_used_data":196476,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93869244,"statfs":{"total":96636764160,"available":96122105856,"internally_reserved":0,"allocated":201191424,"data_stored":200943239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.363}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.441}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":200552448,"data_stored":200548969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148054016,"data_stored":148001238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211959808,"data_stored":211952257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:48.864 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:49.580 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:49.586 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:49.600 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-09-18T12:02:49.233190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":182534239,"num_objects":44584,"num_object_clones":0,"num_object_copies":89168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44584,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":250635,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11575,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160943,"ondisk_log_size":160943,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1447116,"kb_used_data":532784,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281668404,"statfs":{"total":289910292480,"available":288428445696,"internally_reserved":0,"allocated":545570816,"data_stored":544776512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,10,4,12,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25862144,"num_objects":-6314,"num_object_clones":0,"num_object_copies":-12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002846"},"pg_stats":[{"pgid":"2.f","version":"43'16793","reported_seq":16868,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.772086+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:48.772086+0000","last_peered":"2024-09-18T12:02:48.772086+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:48.772086+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11235328,"num_objects":2743,"num_object_clones":0,"num_object_copies":5486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16793,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-856}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16680","reported_seq":16842,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:48.743345+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:48.743345+0000","last_peered":"2024-09-18T12:02:48.743345+0000","last_clean":"2024-09-18T12:02:48.743345+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:48.743345+0000","last_undegraded":"2024-09-18T12:02:48.743345+0000","last_fullsized":"2024-09-18T12:02:48.743345+0000","mapping_epoch":29,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11272192,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13872,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16725","reported_seq":16915,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.747491+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:48.747491+0000","last_peered":"2024-09-18T12:02:48.747491+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:48.747491+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599896,"num_objects":2833,"num_object_clones":0,"num_object_copies":5666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13583,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16510","reported_seq":16543,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:48.743474+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:48.743474+0000","last_peered":"2024-09-18T12:02:48.743474+0000","last_clean":"2024-09-18T12:02:48.743474+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:48.743474+0000","last_undegraded":"2024-09-18T12:02:48.743474+0000","last_fullsized":"2024-09-18T12:02:48.743474+0000","mapping_epoch":29,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11264000,"num_objects":2750,"num_object_clones":0,"num_object_copies":5500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16510,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16686","reported_seq":16881,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:48.840692+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:48.840692+0000","last_peered":"2024-09-18T12:02:48.840692+0000","last_clean":"2024-09-18T12:02:48.840692+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:48.840692+0000","last_undegraded":"2024-09-18T12:02:48.840692+0000","last_fullsized":"2024-09-18T12:02:48.840692+0000","mapping_epoch":29,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11309056,"num_objects":2761,"num_object_clones":0,"num_object_copies":5522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13851,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16599","reported_seq":16684,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.841993+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:48.841993+0000","last_peered":"2024-09-18T12:02:48.841993+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:48.841993+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11568338,"num_objects":2826,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16597,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16656","reported_seq":16744,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.817746+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:48.817746+0000","last_peered":"2024-09-18T12:02:48.817746+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:48.817746+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11125970,"num_objects":2718,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16654,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16827","reported_seq":17056,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:48.707586+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:48.707586+0000","last_peered":"2024-09-18T12:02:48.707586+0000","last_clean":"2024-09-18T12:02:48.707586+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:48.707586+0000","last_undegraded":"2024-09-18T12:02:48.707586+0000","last_fullsized":"2024-09-18T12:02:48.707586+0000","mapping_epoch":29,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11366400,"num_objects":2775,"num_object_clones":0,"num_object_copies":5550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13144,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16510","reported_seq":23642,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:45.236764+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:45.236764+0000","last_peered":"2024-09-18T12:02:45.236764+0000","last_clean":"2024-09-18T12:02:45.236764+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:45.236764+0000","last_undegraded":"2024-09-18T12:02:45.236764+0000","last_fullsized":"2024-09-18T12:02:45.236764+0000","mapping_epoch":38,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11859154,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16508,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16698","reported_seq":18681,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.868074+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:48.868074+0000","last_peered":"2024-09-18T12:02:48.868074+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:48.868074+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11257042,"num_objects":2750,"num_object_clones":0,"num_object_copies":5500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16696,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":166}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16705","reported_seq":16764,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.698518+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:48.698518+0000","last_peered":"2024-09-18T12:02:48.698518+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:48.698518+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11298002,"num_objects":2760,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16703,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-790}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16680","reported_seq":24093,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:48.704388+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:48.704388+0000","last_peered":"2024-09-18T12:02:48.704388+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:48.704388+0000","last_undegraded":"2024-09-18T12:02:48.704388+0000","last_fullsized":"2024-09-18T12:02:48.704388+0000","mapping_epoch":42,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11542528,"num_objects":2818,"num_object_clones":0,"num_object_copies":5636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13160,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2818}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16660","reported_seq":16746,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.786667+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:48.786667+0000","last_peered":"2024-09-18T12:02:48.786667+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:48.786667+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11747945,"num_objects":2869,"num_object_clones":0,"num_object_copies":5738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16659,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-666}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16687","reported_seq":23836,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:45.278437+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:45.278437+0000","last_peered":"2024-09-18T12:02:45.278437+0000","last_clean":"2024-09-18T12:02:45.278437+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:45.278437+0000","last_undegraded":"2024-09-18T12:02:45.278437+0000","last_fullsized":"2024-09-18T12:02:45.278437+0000","mapping_epoch":40,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11614011,"num_objects":2838,"num_object_clones":0,"num_object_copies":5676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16684,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16473","reported_seq":16558,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.635133+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:48.635133+0000","last_peered":"2024-09-18T12:02:48.635133+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:48.635133+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11162834,"num_objects":2727,"num_object_clones":0,"num_object_copies":5454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16471,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-866}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16751","reported_seq":16840,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.633544+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:48.633544+0000","last_peered":"2024-09-18T12:02:48.633544+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:48.633544+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11311524,"num_objects":2765,"num_object_clones":0,"num_object_copies":5530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16748,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-758}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":182534220,"num_objects":44582,"num_object_clones":0,"num_object_copies":89164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":250633,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11574,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":542441472,"data_stored":542377664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160940,"ondisk_log_size":160940,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051691,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509416,"kb_used_data":203232,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93862424,"statfs":{"total":96636764160,"available":96115122176,"internally_reserved":0,"allocated":208109568,"data_stored":207860933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.222},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.27100000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.23999999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.36199999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855453,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437580,"kb_used_data":139712,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93934260,"statfs":{"total":96636764160,"available":96188682240,"internally_reserved":0,"allocated":143065088,"data_stored":142767604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,0,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":2.879}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":5.3620000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.32200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500120,"kb_used_data":189840,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93871720,"statfs":{"total":96636764160,"available":96124641280,"internally_reserved":0,"allocated":194396160,"data_stored":194147975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":193777664,"data_stored":193774185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":141328384,"data_stored":141275606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207335424,"data_stored":207327873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:49.601 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:50.316 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:50.328 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-09-18T12:02:49.233190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":182534239,"num_objects":44584,"num_object_clones":0,"num_object_copies":89168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44584,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":250635,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11575,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160943,"ondisk_log_size":160943,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1447116,"kb_used_data":532784,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281668404,"statfs":{"total":289910292480,"available":288428445696,"internally_reserved":0,"allocated":545570816,"data_stored":544776512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,10,4,12,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25862144,"num_objects":-6314,"num_object_clones":0,"num_object_copies":-12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002846"},"pg_stats":[{"pgid":"2.f","version":"43'16793","reported_seq":16868,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.772086+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:48.772086+0000","last_peered":"2024-09-18T12:02:48.772086+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:48.772086+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11235328,"num_objects":2743,"num_object_clones":0,"num_object_copies":5486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16793,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-856}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16680","reported_seq":16842,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:48.743345+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:48.743345+0000","last_peered":"2024-09-18T12:02:48.743345+0000","last_clean":"2024-09-18T12:02:48.743345+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:48.743345+0000","last_undegraded":"2024-09-18T12:02:48.743345+0000","last_fullsized":"2024-09-18T12:02:48.743345+0000","mapping_epoch":29,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11272192,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13872,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16725","reported_seq":16915,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.747491+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:48.747491+0000","last_peered":"2024-09-18T12:02:48.747491+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:48.747491+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599896,"num_objects":2833,"num_object_clones":0,"num_object_copies":5666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13583,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16510","reported_seq":16543,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:48.743474+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:48.743474+0000","last_peered":"2024-09-18T12:02:48.743474+0000","last_clean":"2024-09-18T12:02:48.743474+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:48.743474+0000","last_undegraded":"2024-09-18T12:02:48.743474+0000","last_fullsized":"2024-09-18T12:02:48.743474+0000","mapping_epoch":29,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11264000,"num_objects":2750,"num_object_clones":0,"num_object_copies":5500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16510,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16686","reported_seq":16881,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:48.840692+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:48.840692+0000","last_peered":"2024-09-18T12:02:48.840692+0000","last_clean":"2024-09-18T12:02:48.840692+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:48.840692+0000","last_undegraded":"2024-09-18T12:02:48.840692+0000","last_fullsized":"2024-09-18T12:02:48.840692+0000","mapping_epoch":29,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11309056,"num_objects":2761,"num_object_clones":0,"num_object_copies":5522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13851,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16599","reported_seq":16684,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.841993+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:48.841993+0000","last_peered":"2024-09-18T12:02:48.841993+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:48.841993+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11568338,"num_objects":2826,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16597,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16656","reported_seq":16744,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.817746+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:48.817746+0000","last_peered":"2024-09-18T12:02:48.817746+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:48.817746+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11125970,"num_objects":2718,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16654,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16827","reported_seq":17056,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:48.707586+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:48.707586+0000","last_peered":"2024-09-18T12:02:48.707586+0000","last_clean":"2024-09-18T12:02:48.707586+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:48.707586+0000","last_undegraded":"2024-09-18T12:02:48.707586+0000","last_fullsized":"2024-09-18T12:02:48.707586+0000","mapping_epoch":29,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11366400,"num_objects":2775,"num_object_clones":0,"num_object_copies":5550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13144,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16510","reported_seq":23642,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:45.236764+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:45.236764+0000","last_peered":"2024-09-18T12:02:45.236764+0000","last_clean":"2024-09-18T12:02:45.236764+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:45.236764+0000","last_undegraded":"2024-09-18T12:02:45.236764+0000","last_fullsized":"2024-09-18T12:02:45.236764+0000","mapping_epoch":38,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11859154,"num_objects":2897,"num_object_clones":0,"num_object_copies":5794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16508,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16698","reported_seq":18681,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.868074+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:48.868074+0000","last_peered":"2024-09-18T12:02:48.868074+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:48.868074+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11257042,"num_objects":2750,"num_object_clones":0,"num_object_copies":5500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16696,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":166}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16705","reported_seq":16764,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.698518+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:48.698518+0000","last_peered":"2024-09-18T12:02:48.698518+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:48.698518+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11298002,"num_objects":2760,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16703,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-790}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16680","reported_seq":24093,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:48.704388+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:48.704388+0000","last_peered":"2024-09-18T12:02:48.704388+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:48.704388+0000","last_undegraded":"2024-09-18T12:02:48.704388+0000","last_fullsized":"2024-09-18T12:02:48.704388+0000","mapping_epoch":42,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11542528,"num_objects":2818,"num_object_clones":0,"num_object_copies":5636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13160,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2818}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16660","reported_seq":16746,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.786667+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:48.786667+0000","last_peered":"2024-09-18T12:02:48.786667+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:48.786667+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11747945,"num_objects":2869,"num_object_clones":0,"num_object_copies":5738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16659,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-666}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16687","reported_seq":23836,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:45.278437+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:45.278437+0000","last_peered":"2024-09-18T12:02:45.278437+0000","last_clean":"2024-09-18T12:02:45.278437+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:45.278437+0000","last_undegraded":"2024-09-18T12:02:45.278437+0000","last_fullsized":"2024-09-18T12:02:45.278437+0000","mapping_epoch":40,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11614011,"num_objects":2838,"num_object_clones":0,"num_object_copies":5676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16684,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16473","reported_seq":16558,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.635133+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:48.635133+0000","last_peered":"2024-09-18T12:02:48.635133+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:48.635133+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11162834,"num_objects":2727,"num_object_clones":0,"num_object_copies":5454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16471,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-866}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16751","reported_seq":16840,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.633544+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:48.633544+0000","last_peered":"2024-09-18T12:02:48.633544+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:48.633544+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11311524,"num_objects":2765,"num_object_clones":0,"num_object_copies":5530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16748,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-758}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":182534220,"num_objects":44582,"num_object_clones":0,"num_object_copies":89164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":250633,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11574,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":542441472,"data_stored":542377664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160940,"ondisk_log_size":160940,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051691,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509416,"kb_used_data":203232,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93862424,"statfs":{"total":96636764160,"available":96115122176,"internally_reserved":0,"allocated":208109568,"data_stored":207860933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.222},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.27100000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.23999999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.36199999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855453,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437580,"kb_used_data":139712,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93934260,"statfs":{"total":96636764160,"available":96188682240,"internally_reserved":0,"allocated":143065088,"data_stored":142767604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,0,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":2.879}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":5.3620000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.32200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500120,"kb_used_data":189840,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93871720,"statfs":{"total":96636764160,"available":96124641280,"internally_reserved":0,"allocated":194396160,"data_stored":194147975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":193777664,"data_stored":193774185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":141328384,"data_stored":141275606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207335424,"data_stored":207327873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:50.330 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:02:50.330 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'16793", 'reported_seq': 16868, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:48.772086+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:02:48.772086+0000', 'last_peered': '2024-09-18T12:02:48.772086+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:02:48.772086+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'6700", 'ondisk_log_start': "37'6700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11235328, 'num_objects': 2743, 'num_object_clones': 0, 'num_object_copies': 5486, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16793, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -856}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:50.330 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:02:50.330 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'16725", 'reported_seq': 16915, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:48.747491+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:02:48.747491+0000', 'last_peered': '2024-09-18T12:02:48.747491+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:02:48.747491+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'6700", 'ondisk_log_start': "37'6700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11599896, 'num_objects': 2833, 'num_object_clones': 0, 'num_object_copies': 5666, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2833, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13583, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -703}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:50.330 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:02:50.330 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'16599", 'reported_seq': 16684, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:48.841993+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:02:48.841993+0000', 'last_peered': '2024-09-18T12:02:48.841993+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:02:48.841993+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'6500", 'ondisk_log_start': "37'6500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11568338, 'num_objects': 2826, 'num_object_clones': 0, 'num_object_copies': 5652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16597, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -701}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:50.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:02:50.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'16656", 'reported_seq': 16744, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:48.817746+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:02:48.817746+0000', 'last_peered': '2024-09-18T12:02:48.817746+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:02:48.817746+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'6600", 'ondisk_log_start': "37'6600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11125970, 'num_objects': 2718, 'num_object_clones': 0, 'num_object_copies': 5436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16654, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -841}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:50.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:02:50.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'16698", 'reported_seq': 18681, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:48.868074+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:02:48.868074+0000', 'last_peered': '2024-09-18T12:02:48.868074+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:02:48.868074+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'6600", 'ondisk_log_start': "37'6600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11257042, 'num_objects': 2750, 'num_object_clones': 0, 'num_object_copies': 5500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2584, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16696, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 952, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 166}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:50.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:02:50.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'16705", 'reported_seq': 16764, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:48.698518+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:02:48.698518+0000', 'last_peered': '2024-09-18T12:02:48.698518+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:02:48.698518+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'6700", 'ondisk_log_start': "37'6700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11298002, 'num_objects': 2760, 'num_object_clones': 0, 'num_object_copies': 5520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16703, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -790}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:50.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:02:50.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'16680", 'reported_seq': 24093, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:02:48.704388+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:02:48.704388+0000', 'last_peered': '2024-09-18T12:02:48.704388+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:02:48.704388+0000', 'last_undegraded': '2024-09-18T12:02:48.704388+0000', 'last_fullsized': '2024-09-18T12:02:48.704388+0000', 'mapping_epoch': 42, 'log_start': "37'6600", 'ondisk_log_start': "37'6600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11542528, 'num_objects': 2818, 'num_object_clones': 0, 'num_object_copies': 5636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13160, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 2818}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:50.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:02:50.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'16660", 'reported_seq': 16746, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:48.786667+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:02:48.786667+0000', 'last_peered': '2024-09-18T12:02:48.786667+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:02:48.786667+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'6600", 'ondisk_log_start': "37'6600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11747945, 'num_objects': 2869, 'num_object_clones': 0, 'num_object_copies': 5738, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16659, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -666}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:50.332 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:02:50.332 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'16473", 'reported_seq': 16558, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:48.635133+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:02:48.635133+0000', 'last_peered': '2024-09-18T12:02:48.635133+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:02:48.635133+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'6400", 'ondisk_log_start': "37'6400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11162834, 'num_objects': 2727, 'num_object_clones': 0, 'num_object_copies': 5454, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2727, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16471, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -866}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:50.332 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:02:50.332 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'16751", 'reported_seq': 16840, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:48.633544+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:02:48.633544+0000', 'last_peered': '2024-09-18T12:02:48.633544+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:02:48.633544+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'6700", 'ondisk_log_start': "37'6700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11311524, 'num_objects': 2765, 'num_object_clones': 0, 'num_object_copies': 5530, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16748, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -758}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:50.332 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:02:51.193 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:51.193 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31019/89168 objects degraded (34.787%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":571,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":44584,"data_bytes":182534239,"bytes_used":1481846784,"bytes_avail":288428445696,"bytes_total":289910292480,"degraded_objects":31019,"degraded_total":89168,"degraded_ratio":0.34787143369818774,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":526},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:02:51.247 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:02:51.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:51.999 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:52.014 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:52.026 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-18T12:02:51.233758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":181334111,"num_objects":44291,"num_object_clones":0,"num_object_copies":88582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":250928,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11575,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160936,"ondisk_log_size":160936,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1448760,"kb_used_data":526108,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281666760,"statfs":{"total":289910292480,"available":288426762240,"internally_reserved":0,"allocated":538734592,"data_stored":537940288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,7,8,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19247104,"num_objects":-4699,"num_object_clones":0,"num_object_copies":-9398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002970"},"pg_stats":[{"pgid":"2.f","version":"43'16793","reported_seq":16868,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.772086+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:48.772086+0000","last_peered":"2024-09-18T12:02:48.772086+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:48.772086+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11235328,"num_objects":2743,"num_object_clones":0,"num_object_copies":5486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16793,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-856}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16680","reported_seq":16842,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:48.743345+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:48.743345+0000","last_peered":"2024-09-18T12:02:48.743345+0000","last_clean":"2024-09-18T12:02:48.743345+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:48.743345+0000","last_undegraded":"2024-09-18T12:02:48.743345+0000","last_fullsized":"2024-09-18T12:02:48.743345+0000","mapping_epoch":29,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11272192,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13872,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16725","reported_seq":16915,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.747491+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:48.747491+0000","last_peered":"2024-09-18T12:02:48.747491+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:48.747491+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599896,"num_objects":2833,"num_object_clones":0,"num_object_copies":5666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2833,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13583,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16510","reported_seq":16543,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:48.743474+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:48.743474+0000","last_peered":"2024-09-18T12:02:48.743474+0000","last_clean":"2024-09-18T12:02:48.743474+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:48.743474+0000","last_undegraded":"2024-09-18T12:02:48.743474+0000","last_fullsized":"2024-09-18T12:02:48.743474+0000","mapping_epoch":29,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11264000,"num_objects":2750,"num_object_clones":0,"num_object_copies":5500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16510,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16686","reported_seq":16881,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:48.840692+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:48.840692+0000","last_peered":"2024-09-18T12:02:48.840692+0000","last_clean":"2024-09-18T12:02:48.840692+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:48.840692+0000","last_undegraded":"2024-09-18T12:02:48.840692+0000","last_fullsized":"2024-09-18T12:02:48.840692+0000","mapping_epoch":29,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11309056,"num_objects":2761,"num_object_clones":0,"num_object_copies":5522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13851,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16599","reported_seq":16684,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.841993+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:48.841993+0000","last_peered":"2024-09-18T12:02:48.841993+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:48.841993+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6500","ondisk_log_start":"37'6500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11568338,"num_objects":2826,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16597,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16656","reported_seq":16744,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.817746+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:48.817746+0000","last_peered":"2024-09-18T12:02:48.817746+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:48.817746+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11125970,"num_objects":2718,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16654,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16827","reported_seq":17056,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:48.707586+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:48.707586+0000","last_peered":"2024-09-18T12:02:48.707586+0000","last_clean":"2024-09-18T12:02:48.707586+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:48.707586+0000","last_undegraded":"2024-09-18T12:02:48.707586+0000","last_fullsized":"2024-09-18T12:02:48.707586+0000","mapping_epoch":29,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11366400,"num_objects":2775,"num_object_clones":0,"num_object_copies":5550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13144,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16672","reported_seq":23804,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:50.164933+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:50.164933+0000","last_peered":"2024-09-18T12:02:50.164933+0000","last_clean":"2024-09-18T12:02:50.164933+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:50.164933+0000","last_undegraded":"2024-09-18T12:02:50.164933+0000","last_fullsized":"2024-09-18T12:02:50.164933+0000","mapping_epoch":38,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11195602,"num_objects":2735,"num_object_clones":0,"num_object_copies":5470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16670,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16698","reported_seq":18681,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.868074+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:48.868074+0000","last_peered":"2024-09-18T12:02:48.868074+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:48.868074+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11257042,"num_objects":2750,"num_object_clones":0,"num_object_copies":5500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16696,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":166}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16705","reported_seq":16764,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.698518+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:48.698518+0000","last_peered":"2024-09-18T12:02:48.698518+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:48.698518+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11298002,"num_objects":2760,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16703,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-790}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16680","reported_seq":24093,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:48.704388+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:48.704388+0000","last_peered":"2024-09-18T12:02:48.704388+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:48.704388+0000","last_undegraded":"2024-09-18T12:02:48.704388+0000","last_fullsized":"2024-09-18T12:02:48.704388+0000","mapping_epoch":42,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11542528,"num_objects":2818,"num_object_clones":0,"num_object_copies":5636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13160,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2818}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16660","reported_seq":16746,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.786667+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:48.786667+0000","last_peered":"2024-09-18T12:02:48.786667+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:48.786667+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11747945,"num_objects":2869,"num_object_clones":0,"num_object_copies":5738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16659,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-666}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16818","reported_seq":23966,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:50.246356+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:50.246356+0000","last_peered":"2024-09-18T12:02:50.246356+0000","last_clean":"2024-09-18T12:02:50.246356+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:50.246356+0000","last_undegraded":"2024-09-18T12:02:50.246356+0000","last_fullsized":"2024-09-18T12:02:50.246356+0000","mapping_epoch":40,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11077435,"num_objects":2707,"num_object_clones":0,"num_object_copies":5414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16815,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16473","reported_seq":16558,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.635133+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:48.635133+0000","last_peered":"2024-09-18T12:02:48.635133+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:48.635133+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6400","ondisk_log_start":"37'6400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11162834,"num_objects":2727,"num_object_clones":0,"num_object_copies":5454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16471,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-866}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16751","reported_seq":16840,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:48.633544+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:48.633544+0000","last_peered":"2024-09-18T12:02:48.633544+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:48.633544+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11311524,"num_objects":2765,"num_object_clones":0,"num_object_copies":5530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16748,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-758}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":181334092,"num_objects":44289,"num_object_clones":0,"num_object_copies":88578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44289,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":250926,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11574,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":535658496,"data_stored":535594688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051691,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509416,"kb_used_data":203232,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93862424,"statfs":{"total":96636764160,"available":96115122176,"internally_reserved":0,"allocated":208109568,"data_stored":207860933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.222},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.27100000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.23999999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.36199999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855454,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439224,"kb_used_data":133036,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93932616,"statfs":{"total":96636764160,"available":96186998784,"internally_reserved":0,"allocated":136228864,"data_stored":135931380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.29399999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.374},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.314}]}]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500120,"kb_used_data":189840,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93871720,"statfs":{"total":96636764160,"available":96124641280,"internally_reserved":0,"allocated":194396160,"data_stored":194147975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":193777664,"data_stored":193774185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":134545408,"data_stored":134492630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207335424,"data_stored":207327873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:55.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:55.755 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:55.767 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-18T12:02:55.234612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":173584479,"num_objects":42399,"num_object_clones":0,"num_object_copies":84798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252820,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11639,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160728,"ondisk_log_size":160728,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1433664,"kb_used_data":515156,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281681856,"statfs":{"total":289910292480,"available":288442220544,"internally_reserved":0,"allocated":527519744,"data_stored":526725440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,10,6,4,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25931776,"num_objects":-6331,"num_object_clones":0,"num_object_copies":-12662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002663"},"pg_stats":[{"pgid":"2.f","version":"43'16918","reported_seq":16993,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.883411+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:53.883411+0000","last_peered":"2024-09-18T12:02:53.883411+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:53.883411+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10723328,"num_objects":2618,"num_object_clones":0,"num_object_copies":5236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16918,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-981}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16801","reported_seq":16962,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:53.719989+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:53.719989+0000","last_peered":"2024-09-18T12:02:53.719989+0000","last_clean":"2024-09-18T12:02:53.719989+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:53.719989+0000","last_undegraded":"2024-09-18T12:02:53.719989+0000","last_fullsized":"2024-09-18T12:02:53.719989+0000","mapping_epoch":29,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10776576,"num_objects":2631,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13993,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16868","reported_seq":17058,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.723714+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:53.723714+0000","last_peered":"2024-09-18T12:02:53.723714+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:53.723714+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11014168,"num_objects":2690,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13726,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-846}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16648","reported_seq":16681,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:53.740996+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:53.740996+0000","last_peered":"2024-09-18T12:02:53.740996+0000","last_clean":"2024-09-18T12:02:53.740996+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:53.740996+0000","last_undegraded":"2024-09-18T12:02:53.740996+0000","last_fullsized":"2024-09-18T12:02:53.740996+0000","mapping_epoch":29,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10698752,"num_objects":2612,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16648,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16818","reported_seq":17013,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:53.739643+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:53.739643+0000","last_peered":"2024-09-18T12:02:53.739643+0000","last_clean":"2024-09-18T12:02:53.739643+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:53.739643+0000","last_undegraded":"2024-09-18T12:02:53.739643+0000","last_fullsized":"2024-09-18T12:02:53.739643+0000","mapping_epoch":29,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10768384,"num_objects":2629,"num_object_clones":0,"num_object_copies":5258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13983,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16742","reported_seq":16819,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.846762+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:53.846762+0000","last_peered":"2024-09-18T12:02:53.846762+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:53.846762+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10982610,"num_objects":2683,"num_object_clones":0,"num_object_copies":5366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16740,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16775","reported_seq":16866,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.797600+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:53.797600+0000","last_peered":"2024-09-18T12:02:53.797600+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:53.797600+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10638546,"num_objects":2599,"num_object_clones":0,"num_object_copies":5198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16773,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-960}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16971","reported_seq":17200,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:53.810425+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:53.810425+0000","last_peered":"2024-09-18T12:02:53.810425+0000","last_clean":"2024-09-18T12:02:53.810425+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:53.810425+0000","last_undegraded":"2024-09-18T12:02:53.810425+0000","last_fullsized":"2024-09-18T12:02:53.810425+0000","mapping_epoch":29,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10776576,"num_objects":2631,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16672","reported_seq":23804,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:50.164933+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:50.164933+0000","last_peered":"2024-09-18T12:02:50.164933+0000","last_clean":"2024-09-18T12:02:50.164933+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:50.164933+0000","last_undegraded":"2024-09-18T12:02:50.164933+0000","last_fullsized":"2024-09-18T12:02:50.164933+0000","mapping_epoch":38,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11195602,"num_objects":2735,"num_object_clones":0,"num_object_copies":5470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16670,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16839","reported_seq":18948,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.832625+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:53.832625+0000","last_peered":"2024-09-18T12:02:53.832625+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:53.832625+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10679506,"num_objects":2609,"num_object_clones":0,"num_object_copies":5218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16837,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":89}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16831","reported_seq":16891,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.752510+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:53.752510+0000","last_peered":"2024-09-18T12:02:53.752510+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:53.752510+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10781906,"num_objects":2634,"num_object_clones":0,"num_object_copies":5268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16829,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-916}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16822","reported_seq":24235,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:53.753921+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:53.753921+0000","last_peered":"2024-09-18T12:02:53.753921+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:53.753921+0000","last_undegraded":"2024-09-18T12:02:53.753921+0000","last_fullsized":"2024-09-18T12:02:53.753921+0000","mapping_epoch":42,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10960896,"num_objects":2676,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13302,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2676}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16799","reported_seq":16885,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.826695+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:53.826695+0000","last_peered":"2024-09-18T12:02:53.826695+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:53.826695+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11178601,"num_objects":2730,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16798,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16818","reported_seq":23966,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:50.246356+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:50.246356+0000","last_peered":"2024-09-18T12:02:50.246356+0000","last_clean":"2024-09-18T12:02:50.246356+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:50.246356+0000","last_undegraded":"2024-09-18T12:02:50.246356+0000","last_fullsized":"2024-09-18T12:02:50.246356+0000","mapping_epoch":40,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11077435,"num_objects":2707,"num_object_clones":0,"num_object_copies":5414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16815,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16617","reported_seq":16702,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.803686+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:53.803686+0000","last_peered":"2024-09-18T12:02:53.803686+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:53.803686+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10573010,"num_objects":2583,"num_object_clones":0,"num_object_copies":5166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16615,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16886","reported_seq":16975,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.810201+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:53.810201+0000","last_peered":"2024-09-18T12:02:53.810201+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:53.810201+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10758564,"num_objects":2630,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16883,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":173584460,"num_objects":42397,"num_object_clones":0,"num_object_copies":84794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42397,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252818,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11638,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":524640256,"data_stored":524576448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160725,"ondisk_log_size":160725,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051692,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509240,"kb_used_data":198884,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93862600,"statfs":{"total":96636764160,"available":96115302400,"internally_reserved":0,"allocated":203657216,"data_stored":203408581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.246}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":2.2599999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855454,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439224,"kb_used_data":133036,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93932616,"statfs":{"total":96636764160,"available":96186998784,"internally_reserved":0,"allocated":136228864,"data_stored":135931380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.29399999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.374},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.314}]}]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485200,"kb_used_data":183236,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93886640,"statfs":{"total":96636764160,"available":96139919360,"internally_reserved":0,"allocated":187633664,"data_stored":187385479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,0,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":5.2229999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187138048,"data_stored":187134569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":134545408,"data_stored":134492630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":202956800,"data_stored":202949249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:55.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:56.337 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:56.362 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:56.381 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-18T12:02:55.234612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":173584479,"num_objects":42399,"num_object_clones":0,"num_object_copies":84798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252820,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11639,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160728,"ondisk_log_size":160728,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1433664,"kb_used_data":515156,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281681856,"statfs":{"total":289910292480,"available":288442220544,"internally_reserved":0,"allocated":527519744,"data_stored":526725440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,10,6,4,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25931776,"num_objects":-6331,"num_object_clones":0,"num_object_copies":-12662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002663"},"pg_stats":[{"pgid":"2.f","version":"43'16918","reported_seq":16993,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.883411+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:53.883411+0000","last_peered":"2024-09-18T12:02:53.883411+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:53.883411+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10723328,"num_objects":2618,"num_object_clones":0,"num_object_copies":5236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16918,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-981}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16801","reported_seq":16962,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:53.719989+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:53.719989+0000","last_peered":"2024-09-18T12:02:53.719989+0000","last_clean":"2024-09-18T12:02:53.719989+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:53.719989+0000","last_undegraded":"2024-09-18T12:02:53.719989+0000","last_fullsized":"2024-09-18T12:02:53.719989+0000","mapping_epoch":29,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10776576,"num_objects":2631,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13993,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16868","reported_seq":17058,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.723714+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:53.723714+0000","last_peered":"2024-09-18T12:02:53.723714+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:53.723714+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11014168,"num_objects":2690,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13726,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-846}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16648","reported_seq":16681,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:53.740996+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:53.740996+0000","last_peered":"2024-09-18T12:02:53.740996+0000","last_clean":"2024-09-18T12:02:53.740996+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:53.740996+0000","last_undegraded":"2024-09-18T12:02:53.740996+0000","last_fullsized":"2024-09-18T12:02:53.740996+0000","mapping_epoch":29,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10698752,"num_objects":2612,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16648,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16818","reported_seq":17013,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:53.739643+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:53.739643+0000","last_peered":"2024-09-18T12:02:53.739643+0000","last_clean":"2024-09-18T12:02:53.739643+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:53.739643+0000","last_undegraded":"2024-09-18T12:02:53.739643+0000","last_fullsized":"2024-09-18T12:02:53.739643+0000","mapping_epoch":29,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10768384,"num_objects":2629,"num_object_clones":0,"num_object_copies":5258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13983,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16742","reported_seq":16819,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.846762+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:53.846762+0000","last_peered":"2024-09-18T12:02:53.846762+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:53.846762+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10982610,"num_objects":2683,"num_object_clones":0,"num_object_copies":5366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16740,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16775","reported_seq":16866,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.797600+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:53.797600+0000","last_peered":"2024-09-18T12:02:53.797600+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:53.797600+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10638546,"num_objects":2599,"num_object_clones":0,"num_object_copies":5198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16773,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-960}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16971","reported_seq":17200,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:53.810425+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:53.810425+0000","last_peered":"2024-09-18T12:02:53.810425+0000","last_clean":"2024-09-18T12:02:53.810425+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:53.810425+0000","last_undegraded":"2024-09-18T12:02:53.810425+0000","last_fullsized":"2024-09-18T12:02:53.810425+0000","mapping_epoch":29,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10776576,"num_objects":2631,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16672","reported_seq":23804,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:50.164933+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:50.164933+0000","last_peered":"2024-09-18T12:02:50.164933+0000","last_clean":"2024-09-18T12:02:50.164933+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:50.164933+0000","last_undegraded":"2024-09-18T12:02:50.164933+0000","last_fullsized":"2024-09-18T12:02:50.164933+0000","mapping_epoch":38,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11195602,"num_objects":2735,"num_object_clones":0,"num_object_copies":5470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16670,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16839","reported_seq":18948,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.832625+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:53.832625+0000","last_peered":"2024-09-18T12:02:53.832625+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:53.832625+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10679506,"num_objects":2609,"num_object_clones":0,"num_object_copies":5218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16837,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":89}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16831","reported_seq":16891,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.752510+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:53.752510+0000","last_peered":"2024-09-18T12:02:53.752510+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:53.752510+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10781906,"num_objects":2634,"num_object_clones":0,"num_object_copies":5268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16829,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-916}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16822","reported_seq":24235,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:53.753921+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:53.753921+0000","last_peered":"2024-09-18T12:02:53.753921+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:53.753921+0000","last_undegraded":"2024-09-18T12:02:53.753921+0000","last_fullsized":"2024-09-18T12:02:53.753921+0000","mapping_epoch":42,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10960896,"num_objects":2676,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13302,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2676}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16799","reported_seq":16885,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.826695+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:53.826695+0000","last_peered":"2024-09-18T12:02:53.826695+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:53.826695+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11178601,"num_objects":2730,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16798,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16818","reported_seq":23966,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:50.246356+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:50.246356+0000","last_peered":"2024-09-18T12:02:50.246356+0000","last_clean":"2024-09-18T12:02:50.246356+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:50.246356+0000","last_undegraded":"2024-09-18T12:02:50.246356+0000","last_fullsized":"2024-09-18T12:02:50.246356+0000","mapping_epoch":40,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11077435,"num_objects":2707,"num_object_clones":0,"num_object_copies":5414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16815,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16617","reported_seq":16702,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.803686+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:53.803686+0000","last_peered":"2024-09-18T12:02:53.803686+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:53.803686+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10573010,"num_objects":2583,"num_object_clones":0,"num_object_copies":5166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16615,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16886","reported_seq":16975,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.810201+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:53.810201+0000","last_peered":"2024-09-18T12:02:53.810201+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:53.810201+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10758564,"num_objects":2630,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16883,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":173584460,"num_objects":42397,"num_object_clones":0,"num_object_copies":84794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42397,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252818,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11638,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":524640256,"data_stored":524576448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160725,"ondisk_log_size":160725,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051692,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509240,"kb_used_data":198884,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93862600,"statfs":{"total":96636764160,"available":96115302400,"internally_reserved":0,"allocated":203657216,"data_stored":203408581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.246}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":2.2599999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855454,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439224,"kb_used_data":133036,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93932616,"statfs":{"total":96636764160,"available":96186998784,"internally_reserved":0,"allocated":136228864,"data_stored":135931380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.29399999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.374},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.314}]}]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485200,"kb_used_data":183236,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93886640,"statfs":{"total":96636764160,"available":96139919360,"internally_reserved":0,"allocated":187633664,"data_stored":187385479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,0,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":5.2229999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187138048,"data_stored":187134569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":134545408,"data_stored":134492630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":202956800,"data_stored":202949249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:02:56.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:02:56.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'16918", 'reported_seq': 16993, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:53.883411+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:02:53.883411+0000', 'last_peered': '2024-09-18T12:02:53.883411+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:02:53.883411+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'6900", 'ondisk_log_start': "37'6900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10723328, 'num_objects': 2618, 'num_object_clones': 0, 'num_object_copies': 5236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2618, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16918, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -981}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:56.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:02:56.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'16868", 'reported_seq': 17058, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:53.723714+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:02:53.723714+0000', 'last_peered': '2024-09-18T12:02:53.723714+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:02:53.723714+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'6800", 'ondisk_log_start': "37'6800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11014168, 'num_objects': 2690, 'num_object_clones': 0, 'num_object_copies': 5380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2690, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13726, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -846}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:56.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:02:56.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'16742", 'reported_seq': 16819, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:53.846762+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:02:53.846762+0000', 'last_peered': '2024-09-18T12:02:53.846762+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:02:53.846762+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'6700", 'ondisk_log_start': "37'6700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10982610, 'num_objects': 2683, 'num_object_clones': 0, 'num_object_copies': 5366, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16740, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -844}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:56.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:02:56.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'16775", 'reported_seq': 16866, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:53.797600+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:02:53.797600+0000', 'last_peered': '2024-09-18T12:02:53.797600+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:02:53.797600+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'6700", 'ondisk_log_start': "37'6700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10638546, 'num_objects': 2599, 'num_object_clones': 0, 'num_object_copies': 5198, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16773, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -960}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:56.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:02:56.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'16839", 'reported_seq': 18948, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:53.832625+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:02:53.832625+0000', 'last_peered': '2024-09-18T12:02:53.832625+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:02:53.832625+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'6800", 'ondisk_log_start': "37'6800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10679506, 'num_objects': 2609, 'num_object_clones': 0, 'num_object_copies': 5218, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2520, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16837, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1016, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 89}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:56.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:02:56.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'16831", 'reported_seq': 16891, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:53.752510+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:02:53.752510+0000', 'last_peered': '2024-09-18T12:02:53.752510+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:02:53.752510+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'6800", 'ondisk_log_start': "37'6800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10781906, 'num_objects': 2634, 'num_object_clones': 0, 'num_object_copies': 5268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16829, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -916}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:56.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:02:56.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'16822", 'reported_seq': 24235, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:02:53.753921+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:02:53.753921+0000', 'last_peered': '2024-09-18T12:02:53.753921+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:02:53.753921+0000', 'last_undegraded': '2024-09-18T12:02:53.753921+0000', 'last_fullsized': '2024-09-18T12:02:53.753921+0000', 'mapping_epoch': 42, 'log_start': "37'6800", 'ondisk_log_start': "37'6800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10960896, 'num_objects': 2676, 'num_object_clones': 0, 'num_object_copies': 5352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2676, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13302, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 2676}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:02:56.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:02:56.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'16799", 'reported_seq': 16885, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:53.826695+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:02:53.826695+0000', 'last_peered': '2024-09-18T12:02:53.826695+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:02:53.826695+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'6700", 'ondisk_log_start': "37'6700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11178601, 'num_objects': 2730, 'num_object_clones': 0, 'num_object_copies': 5460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16798, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -805}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:56.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:02:56.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'16617", 'reported_seq': 16702, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:53.803686+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:02:53.803686+0000', 'last_peered': '2024-09-18T12:02:53.803686+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:02:53.803686+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'6600", 'ondisk_log_start': "37'6600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10573010, 'num_objects': 2583, 'num_object_clones': 0, 'num_object_copies': 5166, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16615, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1010}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:56.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:02:56.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'16886", 'reported_seq': 16975, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:53.810201+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:02:53.810201+0000', 'last_peered': '2024-09-18T12:02:53.810201+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:02:53.810201+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'6800", 'ondisk_log_start': "37'6800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10758564, 'num_objects': 2630, 'num_object_clones': 0, 'num_object_copies': 5260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16883, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -893}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:02:56.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:02:57.334 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:57.334 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30955/84798 objects degraded (36.504%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":577,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":42399,"data_bytes":173584479,"bytes_used":1468071936,"bytes_avail":288442220544,"bytes_total":289910292480,"degraded_objects":30955,"degraded_total":84798,"degraded_ratio":0.36504398688648315,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":221,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":527},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:02:57.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:02:57.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:02:57.989 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:02:58.026 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:02:58.032 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-18T12:02:57.235130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":172408927,"num_objects":42112,"num_object_clones":0,"num_object_copies":84224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42112,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253107,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11639,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1429868,"kb_used_data":507200,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281685652,"statfs":{"total":289910292480,"available":288446107648,"internally_reserved":0,"allocated":519372800,"data_stored":518578496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,9,2,0,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19185664,"num_objects":-4684,"num_object_clones":0,"num_object_copies":-9368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4684,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002766"},"pg_stats":[{"pgid":"2.f","version":"43'16918","reported_seq":16993,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.883411+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:53.883411+0000","last_peered":"2024-09-18T12:02:53.883411+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:53.883411+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10723328,"num_objects":2618,"num_object_clones":0,"num_object_copies":5236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16918,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-981}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16801","reported_seq":16962,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:53.719989+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:53.719989+0000","last_peered":"2024-09-18T12:02:53.719989+0000","last_clean":"2024-09-18T12:02:53.719989+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:53.719989+0000","last_undegraded":"2024-09-18T12:02:53.719989+0000","last_fullsized":"2024-09-18T12:02:53.719989+0000","mapping_epoch":29,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10776576,"num_objects":2631,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13993,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'16868","reported_seq":17058,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.723714+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:53.723714+0000","last_peered":"2024-09-18T12:02:53.723714+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:53.723714+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11014168,"num_objects":2690,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13726,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-846}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16648","reported_seq":16681,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:53.740996+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:53.740996+0000","last_peered":"2024-09-18T12:02:53.740996+0000","last_clean":"2024-09-18T12:02:53.740996+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:53.740996+0000","last_undegraded":"2024-09-18T12:02:53.740996+0000","last_fullsized":"2024-09-18T12:02:53.740996+0000","mapping_epoch":29,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10698752,"num_objects":2612,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16648,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16818","reported_seq":17013,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:53.739643+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:53.739643+0000","last_peered":"2024-09-18T12:02:53.739643+0000","last_clean":"2024-09-18T12:02:53.739643+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:53.739643+0000","last_undegraded":"2024-09-18T12:02:53.739643+0000","last_fullsized":"2024-09-18T12:02:53.739643+0000","mapping_epoch":29,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10768384,"num_objects":2629,"num_object_clones":0,"num_object_copies":5258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13983,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16742","reported_seq":16819,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.846762+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:53.846762+0000","last_peered":"2024-09-18T12:02:53.846762+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:53.846762+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10982610,"num_objects":2683,"num_object_clones":0,"num_object_copies":5366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16740,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16775","reported_seq":16866,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.797600+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:53.797600+0000","last_peered":"2024-09-18T12:02:53.797600+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:53.797600+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10638546,"num_objects":2599,"num_object_clones":0,"num_object_copies":5198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16773,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-960}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'16971","reported_seq":17200,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:53.810425+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:53.810425+0000","last_peered":"2024-09-18T12:02:53.810425+0000","last_clean":"2024-09-18T12:02:53.810425+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:53.810425+0000","last_undegraded":"2024-09-18T12:02:53.810425+0000","last_fullsized":"2024-09-18T12:02:53.810425+0000","mapping_epoch":29,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10776576,"num_objects":2631,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13288,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16810","reported_seq":23942,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:55.349510+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:02:55.349510+0000","last_peered":"2024-09-18T12:02:55.349510+0000","last_clean":"2024-09-18T12:02:55.349510+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:02:55.349510+0000","last_undegraded":"2024-09-18T12:02:55.349510+0000","last_fullsized":"2024-09-18T12:02:55.349510+0000","mapping_epoch":38,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10630354,"num_objects":2597,"num_object_clones":0,"num_object_copies":5194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16808,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16839","reported_seq":18948,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.832625+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:53.832625+0000","last_peered":"2024-09-18T12:02:53.832625+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:53.832625+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10679506,"num_objects":2609,"num_object_clones":0,"num_object_copies":5218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16837,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":89}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16831","reported_seq":16891,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.752510+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:53.752510+0000","last_peered":"2024-09-18T12:02:53.752510+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:53.752510+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10781906,"num_objects":2634,"num_object_clones":0,"num_object_copies":5268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16829,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-916}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16822","reported_seq":24235,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:53.753921+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:53.753921+0000","last_peered":"2024-09-18T12:02:53.753921+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:53.753921+0000","last_undegraded":"2024-09-18T12:02:53.753921+0000","last_fullsized":"2024-09-18T12:02:53.753921+0000","mapping_epoch":42,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10960896,"num_objects":2676,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13302,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2676}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16799","reported_seq":16885,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.826695+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:53.826695+0000","last_peered":"2024-09-18T12:02:53.826695+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:53.826695+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11178601,"num_objects":2730,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16798,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'16967","reported_seq":24117,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:55.271767+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:02:55.271767+0000","last_peered":"2024-09-18T12:02:55.271767+0000","last_clean":"2024-09-18T12:02:55.271767+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:02:55.271767+0000","last_undegraded":"2024-09-18T12:02:55.271767+0000","last_fullsized":"2024-09-18T12:02:55.271767+0000","mapping_epoch":40,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10467131,"num_objects":2558,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16964,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16617","reported_seq":16702,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.803686+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:53.803686+0000","last_peered":"2024-09-18T12:02:53.803686+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:53.803686+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6600","ondisk_log_start":"37'6600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10573010,"num_objects":2583,"num_object_clones":0,"num_object_copies":5166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16615,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'16886","reported_seq":16975,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:53.810201+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:53.810201+0000","last_peered":"2024-09-18T12:02:53.810201+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:53.810201+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10758564,"num_objects":2630,"num_object_clones":0,"num_object_copies":5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16883,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":172408908,"num_objects":42110,"num_object_clones":0,"num_object_copies":84220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42110,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253105,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11638,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":517754880,"data_stored":517691072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051692,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509240,"kb_used_data":198884,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93862600,"statfs":{"total":96636764160,"available":96115302400,"internally_reserved":0,"allocated":203657216,"data_stored":203408581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.246}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":2.2599999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855455,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":435428,"kb_used_data":125080,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93936412,"statfs":{"total":96636764160,"available":96190885888,"internally_reserved":0,"allocated":128081920,"data_stored":127784436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.26400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485200,"kb_used_data":183236,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93886640,"statfs":{"total":96636764160,"available":96139919360,"internally_reserved":0,"allocated":187633664,"data_stored":187385479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,0,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":5.2229999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187138048,"data_stored":187134569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":127660032,"data_stored":127607254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":202956800,"data_stored":202949249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:01.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:01.854 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:01.878 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:01.893 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-09-18T12:03:01.236059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":163721311,"num_objects":39991,"num_object_clones":0,"num_object_copies":79982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39991,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":255228,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11720,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1397432,"kb_used_data":491416,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281718088,"statfs":{"total":289910292480,"available":288479322112,"internally_reserved":0,"allocated":503209984,"data_stored":502415680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,6,1,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18812928,"num_objects":-4593,"num_object_clones":0,"num_object_copies":-9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002869"},"pg_stats":[{"pgid":"2.f","version":"43'17052","reported_seq":17124,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.856763+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:58.856763+0000","last_peered":"2024-09-18T12:02:58.856763+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:58.856763+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10174464,"num_objects":2484,"num_object_clones":0,"num_object_copies":4968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17052,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1115}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16919","reported_seq":17080,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:58.753157+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:58.753157+0000","last_peered":"2024-09-18T12:02:58.753157+0000","last_clean":"2024-09-18T12:02:58.753157+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:58.753157+0000","last_undegraded":"2024-09-18T12:02:58.753157+0000","last_fullsized":"2024-09-18T12:02:58.753157+0000","mapping_epoch":29,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10293248,"num_objects":2513,"num_object_clones":0,"num_object_copies":5026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14111,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17006","reported_seq":17195,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.720563+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:58.720563+0000","last_peered":"2024-09-18T12:02:58.720563+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:58.720563+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10448920,"num_objects":2552,"num_object_clones":0,"num_object_copies":5104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13864,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-984}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16765","reported_seq":16798,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:58.677396+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:58.677396+0000","last_peered":"2024-09-18T12:02:58.677396+0000","last_clean":"2024-09-18T12:02:58.677396+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:58.677396+0000","last_undegraded":"2024-09-18T12:02:58.677396+0000","last_fullsized":"2024-09-18T12:02:58.677396+0000","mapping_epoch":29,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10219520,"num_objects":2495,"num_object_clones":0,"num_object_copies":4990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16765,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16941","reported_seq":17136,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:58.867343+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:58.867343+0000","last_peered":"2024-09-18T12:02:58.867343+0000","last_clean":"2024-09-18T12:02:58.867343+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:58.867343+0000","last_undegraded":"2024-09-18T12:02:58.867343+0000","last_fullsized":"2024-09-18T12:02:58.867343+0000","mapping_epoch":29,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10264576,"num_objects":2506,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14106,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16882","reported_seq":16967,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.711889+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:58.711889+0000","last_peered":"2024-09-18T12:02:58.711889+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:58.711889+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10409170,"num_objects":2543,"num_object_clones":0,"num_object_copies":5086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16880,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-984}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16914","reported_seq":17005,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.821992+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:58.821992+0000","last_peered":"2024-09-18T12:02:58.821992+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:58.821992+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10069202,"num_objects":2460,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16912,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17107","reported_seq":17336,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:58.851979+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:58.851979+0000","last_peered":"2024-09-18T12:02:58.851979+0000","last_clean":"2024-09-18T12:02:58.851979+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:58.851979+0000","last_undegraded":"2024-09-18T12:02:58.851979+0000","last_fullsized":"2024-09-18T12:02:58.851979+0000","mapping_epoch":29,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10219520,"num_objects":2495,"num_object_clones":0,"num_object_copies":4990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13424,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16950","reported_seq":24082,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:00.297780+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:00.297780+0000","last_peered":"2024-09-18T12:03:00.297780+0000","last_clean":"2024-09-18T12:03:00.297780+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:00.297780+0000","last_undegraded":"2024-09-18T12:03:00.297780+0000","last_fullsized":"2024-09-18T12:03:00.297780+0000","mapping_epoch":38,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10056914,"num_objects":2457,"num_object_clones":0,"num_object_copies":4914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16948,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16965","reported_seq":19238,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.852628+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:58.852628+0000","last_peered":"2024-09-18T12:02:58.852628+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:58.852628+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10163410,"num_objects":2483,"num_object_clones":0,"num_object_copies":4966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16963,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1097,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":44}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16990","reported_seq":17049,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.757574+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:58.757574+0000","last_peered":"2024-09-18T12:02:58.757574+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:58.757574+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10130642,"num_objects":2475,"num_object_clones":0,"num_object_copies":4950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16988,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16940","reported_seq":24351,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:58.758839+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:58.758839+0000","last_peered":"2024-09-18T12:02:58.758839+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:58.758839+0000","last_undegraded":"2024-09-18T12:02:58.758839+0000","last_fullsized":"2024-09-18T12:02:58.758839+0000","mapping_epoch":42,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10477568,"num_objects":2558,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13420,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2558}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16937","reported_seq":17020,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.878540+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:58.878540+0000","last_peered":"2024-09-18T12:02:58.878540+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:58.878540+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10613353,"num_objects":2592,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16936,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17094","reported_seq":24241,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:00.472288+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:00.472288+0000","last_peered":"2024-09-18T12:03:00.472288+0000","last_clean":"2024-09-18T12:03:00.472288+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:00.472288+0000","last_undegraded":"2024-09-18T12:03:00.472288+0000","last_fullsized":"2024-09-18T12:03:00.472288+0000","mapping_epoch":40,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9946939,"num_objects":2431,"num_object_clones":0,"num_object_copies":4862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17091,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16749","reported_seq":16834,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.766370+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:58.766370+0000","last_peered":"2024-09-18T12:02:58.766370+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:58.766370+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10032338,"num_objects":2451,"num_object_clones":0,"num_object_copies":4902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16747,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17022","reported_seq":17111,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.851526+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:58.851526+0000","last_peered":"2024-09-18T12:02:58.851526+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:58.851526+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10201508,"num_objects":2494,"num_object_clones":0,"num_object_copies":4988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17019,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":163721292,"num_objects":39989,"num_object_clones":0,"num_object_copies":79978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39989,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":255226,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11719,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":500293632,"data_stored":500229824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160733,"ondisk_log_size":160733,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051693,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492608,"kb_used_data":194744,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93879232,"statfs":{"total":96636764160,"available":96132333568,"internally_reserved":0,"allocated":199417856,"data_stored":199169221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.504},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.48499999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.255},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.27800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855456,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421832,"kb_used_data":119800,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93950008,"statfs":{"total":96636764160,"available":96204808192,"internally_reserved":0,"allocated":122675200,"data_stored":122377716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.45500000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482992,"kb_used_data":176872,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93888848,"statfs":{"total":96636764160,"available":96142180352,"internally_reserved":0,"allocated":181116928,"data_stored":180868743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,0,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":5.2229999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":180736000,"data_stored":180732521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120950784,"data_stored":120898006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":198606848,"data_stored":198599297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:01.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:02.421 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:02.425 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:02.439 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-09-18T12:03:01.236059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":163721311,"num_objects":39991,"num_object_clones":0,"num_object_copies":79982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39991,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":255228,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11720,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1397432,"kb_used_data":491416,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281718088,"statfs":{"total":289910292480,"available":288479322112,"internally_reserved":0,"allocated":503209984,"data_stored":502415680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,6,1,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18812928,"num_objects":-4593,"num_object_clones":0,"num_object_copies":-9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002869"},"pg_stats":[{"pgid":"2.f","version":"43'17052","reported_seq":17124,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.856763+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:58.856763+0000","last_peered":"2024-09-18T12:02:58.856763+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:58.856763+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10174464,"num_objects":2484,"num_object_clones":0,"num_object_copies":4968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17052,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1115}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16919","reported_seq":17080,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:58.753157+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:58.753157+0000","last_peered":"2024-09-18T12:02:58.753157+0000","last_clean":"2024-09-18T12:02:58.753157+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:58.753157+0000","last_undegraded":"2024-09-18T12:02:58.753157+0000","last_fullsized":"2024-09-18T12:02:58.753157+0000","mapping_epoch":29,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10293248,"num_objects":2513,"num_object_clones":0,"num_object_copies":5026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14111,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17006","reported_seq":17195,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.720563+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:58.720563+0000","last_peered":"2024-09-18T12:02:58.720563+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:58.720563+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10448920,"num_objects":2552,"num_object_clones":0,"num_object_copies":5104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13864,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-984}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16765","reported_seq":16798,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:58.677396+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:58.677396+0000","last_peered":"2024-09-18T12:02:58.677396+0000","last_clean":"2024-09-18T12:02:58.677396+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:58.677396+0000","last_undegraded":"2024-09-18T12:02:58.677396+0000","last_fullsized":"2024-09-18T12:02:58.677396+0000","mapping_epoch":29,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10219520,"num_objects":2495,"num_object_clones":0,"num_object_copies":4990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16765,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16941","reported_seq":17136,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:58.867343+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:58.867343+0000","last_peered":"2024-09-18T12:02:58.867343+0000","last_clean":"2024-09-18T12:02:58.867343+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:58.867343+0000","last_undegraded":"2024-09-18T12:02:58.867343+0000","last_fullsized":"2024-09-18T12:02:58.867343+0000","mapping_epoch":29,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10264576,"num_objects":2506,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14106,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16882","reported_seq":16967,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.711889+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:58.711889+0000","last_peered":"2024-09-18T12:02:58.711889+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:58.711889+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10409170,"num_objects":2543,"num_object_clones":0,"num_object_copies":5086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16880,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-984}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16914","reported_seq":17005,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.821992+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:58.821992+0000","last_peered":"2024-09-18T12:02:58.821992+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:58.821992+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10069202,"num_objects":2460,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16912,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17107","reported_seq":17336,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:58.851979+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:58.851979+0000","last_peered":"2024-09-18T12:02:58.851979+0000","last_clean":"2024-09-18T12:02:58.851979+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:58.851979+0000","last_undegraded":"2024-09-18T12:02:58.851979+0000","last_fullsized":"2024-09-18T12:02:58.851979+0000","mapping_epoch":29,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10219520,"num_objects":2495,"num_object_clones":0,"num_object_copies":4990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13424,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16950","reported_seq":24082,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:00.297780+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:00.297780+0000","last_peered":"2024-09-18T12:03:00.297780+0000","last_clean":"2024-09-18T12:03:00.297780+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:00.297780+0000","last_undegraded":"2024-09-18T12:03:00.297780+0000","last_fullsized":"2024-09-18T12:03:00.297780+0000","mapping_epoch":38,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10056914,"num_objects":2457,"num_object_clones":0,"num_object_copies":4914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16948,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16965","reported_seq":19238,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.852628+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:58.852628+0000","last_peered":"2024-09-18T12:02:58.852628+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:58.852628+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10163410,"num_objects":2483,"num_object_clones":0,"num_object_copies":4966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16963,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1097,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":44}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16990","reported_seq":17049,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.757574+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:58.757574+0000","last_peered":"2024-09-18T12:02:58.757574+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:58.757574+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10130642,"num_objects":2475,"num_object_clones":0,"num_object_copies":4950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16988,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16940","reported_seq":24351,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:58.758839+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:58.758839+0000","last_peered":"2024-09-18T12:02:58.758839+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:58.758839+0000","last_undegraded":"2024-09-18T12:02:58.758839+0000","last_fullsized":"2024-09-18T12:02:58.758839+0000","mapping_epoch":42,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10477568,"num_objects":2558,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13420,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2558}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16937","reported_seq":17020,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.878540+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:58.878540+0000","last_peered":"2024-09-18T12:02:58.878540+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:58.878540+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10613353,"num_objects":2592,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16936,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17094","reported_seq":24241,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:00.472288+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:00.472288+0000","last_peered":"2024-09-18T12:03:00.472288+0000","last_clean":"2024-09-18T12:03:00.472288+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:00.472288+0000","last_undegraded":"2024-09-18T12:03:00.472288+0000","last_fullsized":"2024-09-18T12:03:00.472288+0000","mapping_epoch":40,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9946939,"num_objects":2431,"num_object_clones":0,"num_object_copies":4862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17091,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16749","reported_seq":16834,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.766370+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:58.766370+0000","last_peered":"2024-09-18T12:02:58.766370+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:58.766370+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10032338,"num_objects":2451,"num_object_clones":0,"num_object_copies":4902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16747,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17022","reported_seq":17111,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.851526+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:58.851526+0000","last_peered":"2024-09-18T12:02:58.851526+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:58.851526+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10201508,"num_objects":2494,"num_object_clones":0,"num_object_copies":4988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17019,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":163721292,"num_objects":39989,"num_object_clones":0,"num_object_copies":79978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39989,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":255226,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11719,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":500293632,"data_stored":500229824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160733,"ondisk_log_size":160733,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051693,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492608,"kb_used_data":194744,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93879232,"statfs":{"total":96636764160,"available":96132333568,"internally_reserved":0,"allocated":199417856,"data_stored":199169221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.504},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.48499999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.255},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.27800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855456,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421832,"kb_used_data":119800,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93950008,"statfs":{"total":96636764160,"available":96204808192,"internally_reserved":0,"allocated":122675200,"data_stored":122377716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.45500000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482992,"kb_used_data":176872,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93888848,"statfs":{"total":96636764160,"available":96142180352,"internally_reserved":0,"allocated":181116928,"data_stored":180868743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,0,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":5.2229999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":180736000,"data_stored":180732521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120950784,"data_stored":120898006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":198606848,"data_stored":198599297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:02.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:03:02.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'17052", 'reported_seq': 17124, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:58.856763+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:02:58.856763+0000', 'last_peered': '2024-09-18T12:02:58.856763+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:02:58.856763+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'7000", 'ondisk_log_start': "37'7000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10174464, 'num_objects': 2484, 'num_object_clones': 0, 'num_object_copies': 4968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17052, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1115}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:02.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:03:02.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'17006", 'reported_seq': 17195, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:58.720563+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:02:58.720563+0000', 'last_peered': '2024-09-18T12:02:58.720563+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:02:58.720563+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'7000", 'ondisk_log_start': "37'7000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10448920, 'num_objects': 2552, 'num_object_clones': 0, 'num_object_copies': 5104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2552, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13864, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -984}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:02.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:03:02.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'16882", 'reported_seq': 16967, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:58.711889+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:02:58.711889+0000', 'last_peered': '2024-09-18T12:02:58.711889+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:02:58.711889+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'6800", 'ondisk_log_start': "37'6800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10409170, 'num_objects': 2543, 'num_object_clones': 0, 'num_object_copies': 5086, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16880, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -984}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:02.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:03:02.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'16914", 'reported_seq': 17005, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:58.821992+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:02:58.821992+0000', 'last_peered': '2024-09-18T12:02:58.821992+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:02:58.821992+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'6900", 'ondisk_log_start': "37'6900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10069202, 'num_objects': 2460, 'num_object_clones': 0, 'num_object_copies': 4920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16912, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:02.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:03:02.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'16965", 'reported_seq': 19238, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:58.852628+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:02:58.852628+0000', 'last_peered': '2024-09-18T12:02:58.852628+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:02:58.852628+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'6900", 'ondisk_log_start': "37'6900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10163410, 'num_objects': 2483, 'num_object_clones': 0, 'num_object_copies': 4966, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2439, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16963, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1097, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 44}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:02.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:03:02.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'16990", 'reported_seq': 17049, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:58.757574+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:02:58.757574+0000', 'last_peered': '2024-09-18T12:02:58.757574+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:02:58.757574+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'6900", 'ondisk_log_start': "37'6900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10130642, 'num_objects': 2475, 'num_object_clones': 0, 'num_object_copies': 4950, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16988, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1075}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:02.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:03:02.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'16940", 'reported_seq': 24351, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:02:58.758839+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:02:58.758839+0000', 'last_peered': '2024-09-18T12:02:58.758839+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:02:58.758839+0000', 'last_undegraded': '2024-09-18T12:02:58.758839+0000', 'last_fullsized': '2024-09-18T12:02:58.758839+0000', 'mapping_epoch': 42, 'log_start': "37'6900", 'ondisk_log_start': "37'6900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10477568, 'num_objects': 2558, 'num_object_clones': 0, 'num_object_copies': 5116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13420, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 2558}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:02.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:03:02.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'16937", 'reported_seq': 17020, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:58.878540+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:02:58.878540+0000', 'last_peered': '2024-09-18T12:02:58.878540+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:02:58.878540+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'6900", 'ondisk_log_start': "37'6900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10613353, 'num_objects': 2592, 'num_object_clones': 0, 'num_object_copies': 5184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2592, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16936, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -943}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:02.444 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:03:02.444 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'16749", 'reported_seq': 16834, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:58.766370+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:02:58.766370+0000', 'last_peered': '2024-09-18T12:02:58.766370+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:02:58.766370+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'6700", 'ondisk_log_start': "37'6700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10032338, 'num_objects': 2451, 'num_object_clones': 0, 'num_object_copies': 4902, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16747, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1142}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:02.444 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:03:02.444 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'17022", 'reported_seq': 17111, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:02:58.851526+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:02:58.851526+0000', 'last_peered': '2024-09-18T12:02:58.851526+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:02:58.851526+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'7000", 'ondisk_log_start': "37'7000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10201508, 'num_objects': 2494, 'num_object_clones': 0, 'num_object_copies': 4988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17019, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1029}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:02.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:03:03.246 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:03.246 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30874/79982 objects degraded (38.601%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":583,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":39991,"data_bytes":163721311,"bytes_used":1430970368,"bytes_avail":288479322112,"bytes_total":289910292480,"degraded_objects":30874,"degraded_total":79982,"degraded_ratio":0.38601185266685006,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":382},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:03:03.293 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:03:03.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:03.900 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:03.925 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:03.945 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-18T12:03:03.239519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":163721311,"num_objects":39991,"num_object_clones":0,"num_object_copies":79982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39991,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":255228,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11720,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1397432,"kb_used_data":491416,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281718088,"statfs":{"total":289910292480,"available":288479322112,"internally_reserved":0,"allocated":503209984,"data_stored":502415680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,7,6,1,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17612800,"num_objects":-4300,"num_object_clones":0,"num_object_copies":-8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4300,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005761"},"pg_stats":[{"pgid":"2.f","version":"43'17052","reported_seq":17124,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.856763+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:02:58.856763+0000","last_peered":"2024-09-18T12:02:58.856763+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:02:58.856763+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10174464,"num_objects":2484,"num_object_clones":0,"num_object_copies":4968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17052,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1115}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'16919","reported_seq":17080,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:58.753157+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:02:58.753157+0000","last_peered":"2024-09-18T12:02:58.753157+0000","last_clean":"2024-09-18T12:02:58.753157+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:02:58.753157+0000","last_undegraded":"2024-09-18T12:02:58.753157+0000","last_fullsized":"2024-09-18T12:02:58.753157+0000","mapping_epoch":29,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10293248,"num_objects":2513,"num_object_clones":0,"num_object_copies":5026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14111,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17006","reported_seq":17195,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.720563+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:02:58.720563+0000","last_peered":"2024-09-18T12:02:58.720563+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:02:58.720563+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10448920,"num_objects":2552,"num_object_clones":0,"num_object_copies":5104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13864,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-984}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16765","reported_seq":16798,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:58.677396+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:02:58.677396+0000","last_peered":"2024-09-18T12:02:58.677396+0000","last_clean":"2024-09-18T12:02:58.677396+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:02:58.677396+0000","last_undegraded":"2024-09-18T12:02:58.677396+0000","last_fullsized":"2024-09-18T12:02:58.677396+0000","mapping_epoch":29,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10219520,"num_objects":2495,"num_object_clones":0,"num_object_copies":4990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16765,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'16941","reported_seq":17136,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:58.867343+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:02:58.867343+0000","last_peered":"2024-09-18T12:02:58.867343+0000","last_clean":"2024-09-18T12:02:58.867343+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:02:58.867343+0000","last_undegraded":"2024-09-18T12:02:58.867343+0000","last_fullsized":"2024-09-18T12:02:58.867343+0000","mapping_epoch":29,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10264576,"num_objects":2506,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14106,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'16882","reported_seq":16967,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.711889+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:02:58.711889+0000","last_peered":"2024-09-18T12:02:58.711889+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:02:58.711889+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10409170,"num_objects":2543,"num_object_clones":0,"num_object_copies":5086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16880,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-984}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'16914","reported_seq":17005,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.821992+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:02:58.821992+0000","last_peered":"2024-09-18T12:02:58.821992+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:02:58.821992+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10069202,"num_objects":2460,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16912,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1099}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17107","reported_seq":17336,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:02:58.851979+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:02:58.851979+0000","last_peered":"2024-09-18T12:02:58.851979+0000","last_clean":"2024-09-18T12:02:58.851979+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:02:58.851979+0000","last_undegraded":"2024-09-18T12:02:58.851979+0000","last_fullsized":"2024-09-18T12:02:58.851979+0000","mapping_epoch":29,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10219520,"num_objects":2495,"num_object_clones":0,"num_object_copies":4990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13424,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'16950","reported_seq":24082,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:00.297780+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:00.297780+0000","last_peered":"2024-09-18T12:03:00.297780+0000","last_clean":"2024-09-18T12:03:00.297780+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:00.297780+0000","last_undegraded":"2024-09-18T12:03:00.297780+0000","last_fullsized":"2024-09-18T12:03:00.297780+0000","mapping_epoch":38,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10056914,"num_objects":2457,"num_object_clones":0,"num_object_copies":4914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16948,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'16965","reported_seq":19238,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.852628+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:02:58.852628+0000","last_peered":"2024-09-18T12:02:58.852628+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:02:58.852628+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10163410,"num_objects":2483,"num_object_clones":0,"num_object_copies":4966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16963,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1097,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":44}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'16990","reported_seq":17049,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.757574+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:02:58.757574+0000","last_peered":"2024-09-18T12:02:58.757574+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:02:58.757574+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10130642,"num_objects":2475,"num_object_clones":0,"num_object_copies":4950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16988,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'16940","reported_seq":24351,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:02:58.758839+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:02:58.758839+0000","last_peered":"2024-09-18T12:02:58.758839+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:02:58.758839+0000","last_undegraded":"2024-09-18T12:02:58.758839+0000","last_fullsized":"2024-09-18T12:02:58.758839+0000","mapping_epoch":42,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10477568,"num_objects":2558,"num_object_clones":0,"num_object_copies":5116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13420,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2558}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'16937","reported_seq":17020,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.878540+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:02:58.878540+0000","last_peered":"2024-09-18T12:02:58.878540+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:02:58.878540+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10613353,"num_objects":2592,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16936,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17094","reported_seq":24241,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:00.472288+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:00.472288+0000","last_peered":"2024-09-18T12:03:00.472288+0000","last_clean":"2024-09-18T12:03:00.472288+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:00.472288+0000","last_undegraded":"2024-09-18T12:03:00.472288+0000","last_fullsized":"2024-09-18T12:03:00.472288+0000","mapping_epoch":40,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9946939,"num_objects":2431,"num_object_clones":0,"num_object_copies":4862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17091,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16749","reported_seq":16834,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.766370+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:02:58.766370+0000","last_peered":"2024-09-18T12:02:58.766370+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:02:58.766370+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6700","ondisk_log_start":"37'6700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10032338,"num_objects":2451,"num_object_clones":0,"num_object_copies":4902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16747,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17022","reported_seq":17111,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:02:58.851526+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:02:58.851526+0000","last_peered":"2024-09-18T12:02:58.851526+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:02:58.851526+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10201508,"num_objects":2494,"num_object_clones":0,"num_object_copies":4988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17019,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":163721292,"num_objects":39989,"num_object_clones":0,"num_object_copies":79978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39989,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":255226,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11719,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":500293632,"data_stored":500229824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160733,"ondisk_log_size":160733,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051693,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492608,"kb_used_data":194744,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93879232,"statfs":{"total":96636764160,"available":96132333568,"internally_reserved":0,"allocated":199417856,"data_stored":199169221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.504},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.48499999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.255},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.27800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855456,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421832,"kb_used_data":119800,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93950008,"statfs":{"total":96636764160,"available":96204808192,"internally_reserved":0,"allocated":122675200,"data_stored":122377716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.45500000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482992,"kb_used_data":176872,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93888848,"statfs":{"total":96636764160,"available":96142180352,"internally_reserved":0,"allocated":181116928,"data_stored":180868743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,0,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":5.2229999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":180736000,"data_stored":180732521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120950784,"data_stored":120898006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":198606848,"data_stored":198599297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:06.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:07.588 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:07.604 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:07.656 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-18T12:03:07.240394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154800223,"num_objects":37813,"num_object_clones":0,"num_object_copies":75626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":257406,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11780,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160814,"ondisk_log_size":160814,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1391296,"kb_used_data":472804,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281724224,"statfs":{"total":289910292480,"available":288485605376,"internally_reserved":0,"allocated":484151296,"data_stored":483356992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,4,2,9,11,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18784256,"num_objects":-4586,"num_object_clones":0,"num_object_copies":-9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005781"},"pg_stats":[{"pgid":"2.f","version":"43'17178","reported_seq":17253,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.795702+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:03.795702+0000","last_peered":"2024-09-18T12:03:03.795702+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:03.795702+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9658368,"num_objects":2358,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17178,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1241}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17067","reported_seq":17229,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:03.730763+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:03.730763+0000","last_peered":"2024-09-18T12:03:03.730763+0000","last_clean":"2024-09-18T12:03:03.730763+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:03.730763+0000","last_undegraded":"2024-09-18T12:03:03.730763+0000","last_fullsized":"2024-09-18T12:03:03.730763+0000","mapping_epoch":29,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9687040,"num_objects":2365,"num_object_clones":0,"num_object_copies":4730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14259,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17139","reported_seq":17329,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.717870+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:03.717870+0000","last_peered":"2024-09-18T12:03:03.717870+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:03.717870+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9904152,"num_objects":2419,"num_object_clones":0,"num_object_copies":4838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13997,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1117}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16903","reported_seq":16936,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:03.685689+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:03.685689+0000","last_peered":"2024-09-18T12:03:03.685689+0000","last_clean":"2024-09-18T12:03:03.685689+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:03.685689+0000","last_undegraded":"2024-09-18T12:03:03.685689+0000","last_fullsized":"2024-09-18T12:03:03.685689+0000","mapping_epoch":29,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9654272,"num_objects":2357,"num_object_clones":0,"num_object_copies":4714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16903,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17092","reported_seq":17287,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:03.839192+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:03.839192+0000","last_peered":"2024-09-18T12:03:03.839192+0000","last_clean":"2024-09-18T12:03:03.839192+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:03.839192+0000","last_undegraded":"2024-09-18T12:03:03.839192+0000","last_fullsized":"2024-09-18T12:03:03.839192+0000","mapping_epoch":29,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9646080,"num_objects":2355,"num_object_clones":0,"num_object_copies":4710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14257,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17024","reported_seq":17106,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.908381+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:03.908381+0000","last_peered":"2024-09-18T12:03:03.908381+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:03.908381+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9827538,"num_objects":2401,"num_object_clones":0,"num_object_copies":4802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17022,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1126}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17036","reported_seq":17127,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.734368+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:03.734368+0000","last_peered":"2024-09-18T12:03:03.734368+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:03.734368+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9569490,"num_objects":2338,"num_object_clones":0,"num_object_copies":4676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17034,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17253","reported_seq":17480,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:03.851956+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:03.851956+0000","last_peered":"2024-09-18T12:03:03.851956+0000","last_clean":"2024-09-18T12:03:03.851956+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:03.851956+0000","last_undegraded":"2024-09-18T12:03:03.851956+0000","last_fullsized":"2024-09-18T12:03:03.851956+0000","mapping_epoch":29,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9621504,"num_objects":2349,"num_object_clones":0,"num_object_copies":4698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13570,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17088","reported_seq":24220,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:05.533933+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:05.533933+0000","last_peered":"2024-09-18T12:03:05.533933+0000","last_clean":"2024-09-18T12:03:05.533933+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:05.533933+0000","last_undegraded":"2024-09-18T12:03:05.533933+0000","last_fullsized":"2024-09-18T12:03:05.533933+0000","mapping_epoch":38,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9491666,"num_objects":2319,"num_object_clones":0,"num_object_copies":4638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17086,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17102","reported_seq":19493,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.794401+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:03.794401+0000","last_peered":"2024-09-18T12:03:03.794401+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:03.794401+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9602258,"num_objects":2346,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17100,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17135","reported_seq":17195,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.717606+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:03.717606+0000","last_peered":"2024-09-18T12:03:03.717606+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:03.717606+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9536722,"num_objects":2330,"num_object_clones":0,"num_object_copies":4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17133,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1220}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17070","reported_seq":24483,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:03.623392+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:03.623392+0000","last_peered":"2024-09-18T12:03:03.623392+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:03.623392+0000","last_undegraded":"2024-09-18T12:03:03.623392+0000","last_fullsized":"2024-09-18T12:03:03.623392+0000","mapping_epoch":42,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9945088,"num_objects":2428,"num_object_clones":0,"num_object_copies":4856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13550,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2428}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17069","reported_seq":17155,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.799464+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:03.799464+0000","last_peered":"2024-09-18T12:03:03.799464+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:03.799464+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10072681,"num_objects":2460,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17068,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17223","reported_seq":24373,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:05.419243+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:05.419243+0000","last_peered":"2024-09-18T12:03:05.419243+0000","last_clean":"2024-09-18T12:03:05.419243+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:05.419243+0000","last_undegraded":"2024-09-18T12:03:05.419243+0000","last_fullsized":"2024-09-18T12:03:05.419243+0000","mapping_epoch":40,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9418555,"num_objects":2302,"num_object_clones":0,"num_object_copies":4604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17220,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16889","reported_seq":16974,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.744930+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:03.744930+0000","last_peered":"2024-09-18T12:03:03.744930+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:03.744930+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9458898,"num_objects":2311,"num_object_clones":0,"num_object_copies":4622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16887,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17143","reported_seq":17232,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.862920+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:03.862920+0000","last_peered":"2024-09-18T12:03:03.862920+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:03.862920+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9705892,"num_objects":2373,"num_object_clones":0,"num_object_copies":4746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17140,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":154800204,"num_objects":37811,"num_object_clones":0,"num_object_copies":75622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37811,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":257404,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11779,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":482390016,"data_stored":482326208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160811,"ondisk_log_size":160811,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051694,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492316,"kb_used_data":190292,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93879524,"statfs":{"total":96636764160,"available":96132632576,"internally_reserved":0,"allocated":194859008,"data_stored":194610373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":2.1320000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.29199999999999998},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.42899999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855457,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418192,"kb_used_data":112008,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93953648,"statfs":{"total":96636764160,"available":96208535552,"internally_reserved":0,"allocated":114696192,"data_stored":114398708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.76600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480788,"kb_used_data":170504,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93891052,"statfs":{"total":96636764160,"available":96144437248,"internally_reserved":0,"allocated":174596096,"data_stored":174347911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,0,2,3,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.216},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":2.1779999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":7.2320000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":174043136,"data_stored":174039657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114290688,"data_stored":114237910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194056192,"data_stored":194048641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:07.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:08.316 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:08.328 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:08.357 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-18T12:03:07.240394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154800223,"num_objects":37813,"num_object_clones":0,"num_object_copies":75626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":257406,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11780,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160814,"ondisk_log_size":160814,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1391296,"kb_used_data":472804,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281724224,"statfs":{"total":289910292480,"available":288485605376,"internally_reserved":0,"allocated":484151296,"data_stored":483356992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,4,2,9,11,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18784256,"num_objects":-4586,"num_object_clones":0,"num_object_copies":-9172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005781"},"pg_stats":[{"pgid":"2.f","version":"43'17178","reported_seq":17253,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.795702+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:03.795702+0000","last_peered":"2024-09-18T12:03:03.795702+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:03.795702+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9658368,"num_objects":2358,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17178,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1241}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17067","reported_seq":17229,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:03.730763+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:03.730763+0000","last_peered":"2024-09-18T12:03:03.730763+0000","last_clean":"2024-09-18T12:03:03.730763+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:03.730763+0000","last_undegraded":"2024-09-18T12:03:03.730763+0000","last_fullsized":"2024-09-18T12:03:03.730763+0000","mapping_epoch":29,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9687040,"num_objects":2365,"num_object_clones":0,"num_object_copies":4730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14259,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17139","reported_seq":17329,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.717870+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:03.717870+0000","last_peered":"2024-09-18T12:03:03.717870+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:03.717870+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9904152,"num_objects":2419,"num_object_clones":0,"num_object_copies":4838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13997,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1117}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'16903","reported_seq":16936,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:03.685689+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:03.685689+0000","last_peered":"2024-09-18T12:03:03.685689+0000","last_clean":"2024-09-18T12:03:03.685689+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:03.685689+0000","last_undegraded":"2024-09-18T12:03:03.685689+0000","last_fullsized":"2024-09-18T12:03:03.685689+0000","mapping_epoch":29,"log_start":"37'6900","ondisk_log_start":"37'6900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9654272,"num_objects":2357,"num_object_clones":0,"num_object_copies":4714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16903,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17092","reported_seq":17287,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:03.839192+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:03.839192+0000","last_peered":"2024-09-18T12:03:03.839192+0000","last_clean":"2024-09-18T12:03:03.839192+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:03.839192+0000","last_undegraded":"2024-09-18T12:03:03.839192+0000","last_fullsized":"2024-09-18T12:03:03.839192+0000","mapping_epoch":29,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9646080,"num_objects":2355,"num_object_clones":0,"num_object_copies":4710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14257,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17024","reported_seq":17106,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.908381+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:03.908381+0000","last_peered":"2024-09-18T12:03:03.908381+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:03.908381+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9827538,"num_objects":2401,"num_object_clones":0,"num_object_copies":4802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17022,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1126}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17036","reported_seq":17127,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.734368+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:03.734368+0000","last_peered":"2024-09-18T12:03:03.734368+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:03.734368+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9569490,"num_objects":2338,"num_object_clones":0,"num_object_copies":4676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17034,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17253","reported_seq":17480,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:03.851956+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:03.851956+0000","last_peered":"2024-09-18T12:03:03.851956+0000","last_clean":"2024-09-18T12:03:03.851956+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:03.851956+0000","last_undegraded":"2024-09-18T12:03:03.851956+0000","last_fullsized":"2024-09-18T12:03:03.851956+0000","mapping_epoch":29,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9621504,"num_objects":2349,"num_object_clones":0,"num_object_copies":4698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13570,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17088","reported_seq":24220,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:05.533933+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:05.533933+0000","last_peered":"2024-09-18T12:03:05.533933+0000","last_clean":"2024-09-18T12:03:05.533933+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:05.533933+0000","last_undegraded":"2024-09-18T12:03:05.533933+0000","last_fullsized":"2024-09-18T12:03:05.533933+0000","mapping_epoch":38,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9491666,"num_objects":2319,"num_object_clones":0,"num_object_copies":4638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17086,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17102","reported_seq":19493,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.794401+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:03.794401+0000","last_peered":"2024-09-18T12:03:03.794401+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:03.794401+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9602258,"num_objects":2346,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17100,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17135","reported_seq":17195,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.717606+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:03.717606+0000","last_peered":"2024-09-18T12:03:03.717606+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:03.717606+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9536722,"num_objects":2330,"num_object_clones":0,"num_object_copies":4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17133,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1220}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17070","reported_seq":24483,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:03.623392+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:03.623392+0000","last_peered":"2024-09-18T12:03:03.623392+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:03.623392+0000","last_undegraded":"2024-09-18T12:03:03.623392+0000","last_fullsized":"2024-09-18T12:03:03.623392+0000","mapping_epoch":42,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9945088,"num_objects":2428,"num_object_clones":0,"num_object_copies":4856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13550,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2428}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17069","reported_seq":17155,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.799464+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:03.799464+0000","last_peered":"2024-09-18T12:03:03.799464+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:03.799464+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10072681,"num_objects":2460,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17068,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17223","reported_seq":24373,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:05.419243+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:05.419243+0000","last_peered":"2024-09-18T12:03:05.419243+0000","last_clean":"2024-09-18T12:03:05.419243+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:05.419243+0000","last_undegraded":"2024-09-18T12:03:05.419243+0000","last_fullsized":"2024-09-18T12:03:05.419243+0000","mapping_epoch":40,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9418555,"num_objects":2302,"num_object_clones":0,"num_object_copies":4604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17220,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'16889","reported_seq":16974,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.744930+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:03.744930+0000","last_peered":"2024-09-18T12:03:03.744930+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:03.744930+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'6800","ondisk_log_start":"37'6800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9458898,"num_objects":2311,"num_object_clones":0,"num_object_copies":4622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16887,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17143","reported_seq":17232,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:03.862920+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:03.862920+0000","last_peered":"2024-09-18T12:03:03.862920+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:03.862920+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9705892,"num_objects":2373,"num_object_clones":0,"num_object_copies":4746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17140,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":154800204,"num_objects":37811,"num_object_clones":0,"num_object_copies":75622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37811,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":257404,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11779,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":482390016,"data_stored":482326208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160811,"ondisk_log_size":160811,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051694,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492316,"kb_used_data":190292,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93879524,"statfs":{"total":96636764160,"available":96132632576,"internally_reserved":0,"allocated":194859008,"data_stored":194610373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":2.1320000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.29199999999999998},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.42899999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855457,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418192,"kb_used_data":112008,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93953648,"statfs":{"total":96636764160,"available":96208535552,"internally_reserved":0,"allocated":114696192,"data_stored":114398708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.76600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480788,"kb_used_data":170504,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93891052,"statfs":{"total":96636764160,"available":96144437248,"internally_reserved":0,"allocated":174596096,"data_stored":174347911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,0,2,3,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.216},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":2.1779999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":7.2320000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":174043136,"data_stored":174039657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114290688,"data_stored":114237910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194056192,"data_stored":194048641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:03:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'17178", 'reported_seq': 17253, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:03.795702+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:03:03.795702+0000', 'last_peered': '2024-09-18T12:03:03.795702+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:03:03.795702+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'7100", 'ondisk_log_start': "37'7100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9658368, 'num_objects': 2358, 'num_object_clones': 0, 'num_object_copies': 4716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17178, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1241}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:03:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'17139", 'reported_seq': 17329, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:03.717870+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:03:03.717870+0000', 'last_peered': '2024-09-18T12:03:03.717870+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:03:03.717870+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'7100", 'ondisk_log_start': "37'7100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9904152, 'num_objects': 2419, 'num_object_clones': 0, 'num_object_copies': 4838, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2419, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13997, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1117}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:03:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'17024", 'reported_seq': 17106, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:03.908381+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:03:03.908381+0000', 'last_peered': '2024-09-18T12:03:03.908381+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:03:03.908381+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'7000", 'ondisk_log_start': "37'7000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9827538, 'num_objects': 2401, 'num_object_clones': 0, 'num_object_copies': 4802, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2401, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17022, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1126}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:03:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'17036", 'reported_seq': 17127, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:03.734368+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:03:03.734368+0000', 'last_peered': '2024-09-18T12:03:03.734368+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:03:03.734368+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'7000", 'ondisk_log_start': "37'7000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9569490, 'num_objects': 2338, 'num_object_clones': 0, 'num_object_copies': 4676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17034, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1221}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:03:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'17102", 'reported_seq': 19493, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:03.794401+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:03:03.794401+0000', 'last_peered': '2024-09-18T12:03:03.794401+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:03:03.794401+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'7100", 'ondisk_log_start': "37'7100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9602258, 'num_objects': 2346, 'num_object_clones': 0, 'num_object_copies': 4692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2379, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17100, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1157, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:03:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'17135", 'reported_seq': 17195, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:03.717606+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:03:03.717606+0000', 'last_peered': '2024-09-18T12:03:03.717606+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:03:03.717606+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'7100", 'ondisk_log_start': "37'7100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9536722, 'num_objects': 2330, 'num_object_clones': 0, 'num_object_copies': 4660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17133, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1220}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:03:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'17070", 'reported_seq': 24483, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:03:03.623392+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:03:03.623392+0000', 'last_peered': '2024-09-18T12:03:03.623392+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:03:03.623392+0000', 'last_undegraded': '2024-09-18T12:03:03.623392+0000', 'last_fullsized': '2024-09-18T12:03:03.623392+0000', 'mapping_epoch': 42, 'log_start': "37'7000", 'ondisk_log_start': "37'7000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9945088, 'num_objects': 2428, 'num_object_clones': 0, 'num_object_copies': 4856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13550, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 2428}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:03:08.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'17069", 'reported_seq': 17155, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:03.799464+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:03:03.799464+0000', 'last_peered': '2024-09-18T12:03:03.799464+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:03:03.799464+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'7000", 'ondisk_log_start': "37'7000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10072681, 'num_objects': 2460, 'num_object_clones': 0, 'num_object_copies': 4920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17068, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1075}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:08.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:03:08.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'16889", 'reported_seq': 16974, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:03.744930+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:03:03.744930+0000', 'last_peered': '2024-09-18T12:03:03.744930+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:03:03.744930+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'6800", 'ondisk_log_start': "37'6800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9458898, 'num_objects': 2311, 'num_object_clones': 0, 'num_object_copies': 4622, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2311, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16887, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1282}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:08.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:03:08.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'17143", 'reported_seq': 17232, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:03.862920+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:03:03.862920+0000', 'last_peered': '2024-09-18T12:03:03.862920+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:03:03.862920+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'7100", 'ondisk_log_start': "37'7100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9705892, 'num_objects': 2373, 'num_object_clones': 0, 'num_object_copies': 4746, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17140, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1150}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:08.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:03:09.183 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:09.183 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30814/75626 objects degraded (40.745%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":589,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":37813,"data_bytes":154800223,"bytes_used":1424687104,"bytes_avail":288485605376,"bytes_total":289910292480,"degraded_objects":30814,"degraded_total":75626,"degraded_ratio":0.4074524634385,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":381},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:03:09.215 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:03:09.216 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:09.839 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:09.871 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-18T12:03:09.240829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147050591,"num_objects":35921,"num_object_clones":0,"num_object_copies":71842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259298,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11860,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160806,"ondisk_log_size":160806,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1376416,"kb_used_data":462076,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281739104,"statfs":{"total":289910292480,"available":288500842496,"internally_reserved":0,"allocated":473165824,"data_stored":472371520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,3,7,11,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25358336,"num_objects":-6191,"num_object_clones":0,"num_object_copies":-12382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005698"},"pg_stats":[{"pgid":"2.f","version":"43'17308","reported_seq":17383,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.614763+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:08.614763+0000","last_peered":"2024-09-18T12:03:08.614763+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:08.614763+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'7300","ondisk_log_start":"37'7300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9125888,"num_objects":2228,"num_object_clones":0,"num_object_copies":4456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17308,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1371}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17192","reported_seq":17354,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:08.678261+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:08.678261+0000","last_peered":"2024-09-18T12:03:08.678261+0000","last_clean":"2024-09-18T12:03:08.678261+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:08.678261+0000","last_undegraded":"2024-09-18T12:03:08.678261+0000","last_fullsized":"2024-09-18T12:03:08.678261+0000","mapping_epoch":29,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9175040,"num_objects":2240,"num_object_clones":0,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14384,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17290","reported_seq":17480,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.709352+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:08.709352+0000","last_peered":"2024-09-18T12:03:08.709352+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:08.709352+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9285656,"num_objects":2268,"num_object_clones":0,"num_object_copies":4536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2268,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14148,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1268}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17042","reported_seq":17070,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:08.729431+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:08.729431+0000","last_peered":"2024-09-18T12:03:08.729431+0000","last_clean":"2024-09-18T12:03:08.729431+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:08.729431+0000","last_undegraded":"2024-09-18T12:03:08.729431+0000","last_fullsized":"2024-09-18T12:03:08.729431+0000","mapping_epoch":29,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9084928,"num_objects":2218,"num_object_clones":0,"num_object_copies":4436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17042,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17204","reported_seq":17399,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:08.927562+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:08.927562+0000","last_peered":"2024-09-18T12:03:08.927562+0000","last_clean":"2024-09-18T12:03:08.927562+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:08.927562+0000","last_undegraded":"2024-09-18T12:03:08.927562+0000","last_fullsized":"2024-09-18T12:03:08.927562+0000","mapping_epoch":29,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9187328,"num_objects":2243,"num_object_clones":0,"num_object_copies":4486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14369,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17160","reported_seq":17244,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.894439+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:08.894439+0000","last_peered":"2024-09-18T12:03:08.894439+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:08.894439+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9270482,"num_objects":2265,"num_object_clones":0,"num_object_copies":4530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17158,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17174","reported_seq":17265,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.727541+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:08.727541+0000","last_peered":"2024-09-18T12:03:08.727541+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:08.727541+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9004242,"num_objects":2200,"num_object_clones":0,"num_object_copies":4400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17172,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17388","reported_seq":17617,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:08.893047+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:08.893047+0000","last_peered":"2024-09-18T12:03:08.893047+0000","last_clean":"2024-09-18T12:03:08.893047+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:08.893047+0000","last_undegraded":"2024-09-18T12:03:08.893047+0000","last_fullsized":"2024-09-18T12:03:08.893047+0000","mapping_epoch":29,"log_start":"37'7300","ondisk_log_start":"37'7300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9068544,"num_objects":2214,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13705,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17088","reported_seq":24220,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:05.533933+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:05.533933+0000","last_peered":"2024-09-18T12:03:05.533933+0000","last_clean":"2024-09-18T12:03:05.533933+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:05.533933+0000","last_undegraded":"2024-09-18T12:03:05.533933+0000","last_fullsized":"2024-09-18T12:03:05.533933+0000","mapping_epoch":38,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9491666,"num_objects":2319,"num_object_clones":0,"num_object_copies":4638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17086,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17235","reported_seq":19788,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.893234+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:08.893234+0000","last_peered":"2024-09-18T12:03:08.893234+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:08.893234+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9057490,"num_objects":2213,"num_object_clones":0,"num_object_copies":4426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17233,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-86}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17293","reported_seq":17352,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.779013+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:08.779013+0000","last_peered":"2024-09-18T12:03:08.779013+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:08.779013+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8889554,"num_objects":2172,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17291,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1378}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17201","reported_seq":24614,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:08.753035+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:08.753035+0000","last_peered":"2024-09-18T12:03:08.753035+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:08.753035+0000","last_undegraded":"2024-09-18T12:03:08.753035+0000","last_fullsized":"2024-09-18T12:03:08.753035+0000","mapping_epoch":42,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9408512,"num_objects":2297,"num_object_clones":0,"num_object_copies":4594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13681,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17203","reported_seq":17289,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.847723+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:08.847723+0000","last_peered":"2024-09-18T12:03:08.847723+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:08.847723+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9523817,"num_objects":2326,"num_object_clones":0,"num_object_copies":4652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17202,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17223","reported_seq":24373,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:05.419243+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:05.419243+0000","last_peered":"2024-09-18T12:03:05.419243+0000","last_clean":"2024-09-18T12:03:05.419243+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:05.419243+0000","last_undegraded":"2024-09-18T12:03:05.419243+0000","last_fullsized":"2024-09-18T12:03:05.419243+0000","mapping_epoch":40,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9418555,"num_objects":2302,"num_object_clones":0,"num_object_copies":4604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17220,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17022","reported_seq":17107,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.835542+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:08.835542+0000","last_peered":"2024-09-18T12:03:08.835542+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:08.835542+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8914130,"num_objects":2178,"num_object_clones":0,"num_object_copies":4356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17020,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17280","reported_seq":17367,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.852394+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:08.852394+0000","last_peered":"2024-09-18T12:03:08.852394+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:08.852394+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9144740,"num_objects":2236,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17277,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":147050572,"num_objects":35919,"num_object_clones":0,"num_object_copies":71838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35919,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259296,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11859,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471334912,"data_stored":471271104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051695,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492100,"kb_used_data":185912,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93879740,"statfs":{"total":96636764160,"available":96132853760,"internally_reserved":0,"allocated":190373888,"data_stored":190125253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.28399999999999997},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.39800000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":2.282}]}]},{"osd":1,"up_from":35,"seq":150323855457,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418192,"kb_used_data":112008,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93953648,"statfs":{"total":96636764160,"available":96208535552,"internally_reserved":0,"allocated":114696192,"data_stored":114398708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.76600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466124,"kb_used_data":164156,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93905716,"statfs":{"total":96636764160,"available":96159453184,"internally_reserved":0,"allocated":168095744,"data_stored":167847559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,0,2,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.26500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.23699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167493632,"data_stored":167490153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114290688,"data_stored":114237910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":189550592,"data_stored":189543041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:12.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:13.630 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:13.646 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:13.668 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-18T12:03:13.241776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":145981535,"num_objects":35660,"num_object_clones":0,"num_object_copies":71320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259559,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11860,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160767,"ondisk_log_size":160767,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1361732,"kb_used_data":455700,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281753788,"statfs":{"total":289910292480,"available":288515878912,"internally_reserved":0,"allocated":466636800,"data_stored":465842496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,3,7,11,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17739776,"num_objects":-4331,"num_object_clones":0,"num_object_copies":-8662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005716"},"pg_stats":[{"pgid":"2.f","version":"43'17308","reported_seq":17383,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.614763+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:08.614763+0000","last_peered":"2024-09-18T12:03:08.614763+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:08.614763+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'7300","ondisk_log_start":"37'7300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9125888,"num_objects":2228,"num_object_clones":0,"num_object_copies":4456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17308,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1371}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17192","reported_seq":17354,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:08.678261+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:08.678261+0000","last_peered":"2024-09-18T12:03:08.678261+0000","last_clean":"2024-09-18T12:03:08.678261+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:08.678261+0000","last_undegraded":"2024-09-18T12:03:08.678261+0000","last_fullsized":"2024-09-18T12:03:08.678261+0000","mapping_epoch":29,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9175040,"num_objects":2240,"num_object_clones":0,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14384,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17290","reported_seq":17480,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.709352+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:08.709352+0000","last_peered":"2024-09-18T12:03:08.709352+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:08.709352+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9285656,"num_objects":2268,"num_object_clones":0,"num_object_copies":4536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2268,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14148,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1268}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17042","reported_seq":17070,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:08.729431+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:08.729431+0000","last_peered":"2024-09-18T12:03:08.729431+0000","last_clean":"2024-09-18T12:03:08.729431+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:08.729431+0000","last_undegraded":"2024-09-18T12:03:08.729431+0000","last_fullsized":"2024-09-18T12:03:08.729431+0000","mapping_epoch":29,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9084928,"num_objects":2218,"num_object_clones":0,"num_object_copies":4436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17042,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17204","reported_seq":17399,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:08.927562+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:08.927562+0000","last_peered":"2024-09-18T12:03:08.927562+0000","last_clean":"2024-09-18T12:03:08.927562+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:08.927562+0000","last_undegraded":"2024-09-18T12:03:08.927562+0000","last_fullsized":"2024-09-18T12:03:08.927562+0000","mapping_epoch":29,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9187328,"num_objects":2243,"num_object_clones":0,"num_object_copies":4486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14369,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17160","reported_seq":17244,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.894439+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:08.894439+0000","last_peered":"2024-09-18T12:03:08.894439+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:08.894439+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9270482,"num_objects":2265,"num_object_clones":0,"num_object_copies":4530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17158,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17174","reported_seq":17265,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.727541+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:08.727541+0000","last_peered":"2024-09-18T12:03:08.727541+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:08.727541+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9004242,"num_objects":2200,"num_object_clones":0,"num_object_copies":4400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17172,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17388","reported_seq":17617,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:08.893047+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:08.893047+0000","last_peered":"2024-09-18T12:03:08.893047+0000","last_clean":"2024-09-18T12:03:08.893047+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:08.893047+0000","last_undegraded":"2024-09-18T12:03:08.893047+0000","last_fullsized":"2024-09-18T12:03:08.893047+0000","mapping_epoch":29,"log_start":"37'7300","ondisk_log_start":"37'7300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9068544,"num_objects":2214,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13705,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17213","reported_seq":24345,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:10.534578+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:10.534578+0000","last_peered":"2024-09-18T12:03:10.534578+0000","last_clean":"2024-09-18T12:03:10.534578+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:10.534578+0000","last_undegraded":"2024-09-18T12:03:10.534578+0000","last_fullsized":"2024-09-18T12:03:10.534578+0000","mapping_epoch":38,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8979666,"num_objects":2194,"num_object_clones":0,"num_object_copies":4388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17211,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17235","reported_seq":19788,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.893234+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:08.893234+0000","last_peered":"2024-09-18T12:03:08.893234+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:08.893234+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9057490,"num_objects":2213,"num_object_clones":0,"num_object_copies":4426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17233,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-86}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17293","reported_seq":17352,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.779013+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:08.779013+0000","last_peered":"2024-09-18T12:03:08.779013+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:08.779013+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8889554,"num_objects":2172,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17291,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1378}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17201","reported_seq":24614,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:08.753035+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:08.753035+0000","last_peered":"2024-09-18T12:03:08.753035+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:08.753035+0000","last_undegraded":"2024-09-18T12:03:08.753035+0000","last_fullsized":"2024-09-18T12:03:08.753035+0000","mapping_epoch":42,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9408512,"num_objects":2297,"num_object_clones":0,"num_object_copies":4594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13681,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17203","reported_seq":17289,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.847723+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:08.847723+0000","last_peered":"2024-09-18T12:03:08.847723+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:08.847723+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9523817,"num_objects":2326,"num_object_clones":0,"num_object_copies":4652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17202,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17359","reported_seq":24509,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:10.571477+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:10.571477+0000","last_peered":"2024-09-18T12:03:10.571477+0000","last_clean":"2024-09-18T12:03:10.571477+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:10.571477+0000","last_undegraded":"2024-09-18T12:03:10.571477+0000","last_fullsized":"2024-09-18T12:03:10.571477+0000","mapping_epoch":40,"log_start":"37'7300","ondisk_log_start":"37'7300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8861499,"num_objects":2166,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17356,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17022","reported_seq":17107,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.835542+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:08.835542+0000","last_peered":"2024-09-18T12:03:08.835542+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:08.835542+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8914130,"num_objects":2178,"num_object_clones":0,"num_object_copies":4356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17020,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17280","reported_seq":17367,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.852394+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:08.852394+0000","last_peered":"2024-09-18T12:03:08.852394+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:08.852394+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9144740,"num_objects":2236,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17277,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":145981516,"num_objects":35658,"num_object_clones":0,"num_object_copies":71316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35658,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259557,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11859,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":464687104,"data_stored":464623296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160764,"ondisk_log_size":160764,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051695,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492100,"kb_used_data":185912,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93879740,"statfs":{"total":96636764160,"available":96132853760,"internally_reserved":0,"allocated":190373888,"data_stored":190125253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.28399999999999997},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.39800000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":2.282}]}]},{"osd":1,"up_from":35,"seq":150323855458,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403508,"kb_used_data":105632,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93968332,"statfs":{"total":96636764160,"available":96223571968,"internally_reserved":0,"allocated":108167168,"data_stored":107869684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.76600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466124,"kb_used_data":164156,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93905716,"statfs":{"total":96636764160,"available":96159453184,"internally_reserved":0,"allocated":168095744,"data_stored":167847559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,0,2,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.26500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.23699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167493632,"data_stored":167490153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107642880,"data_stored":107590102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":189550592,"data_stored":189543041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:13.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:14.287 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:14.318 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:14.359 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-18T12:03:13.241776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":145981535,"num_objects":35660,"num_object_clones":0,"num_object_copies":71320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259559,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11860,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160767,"ondisk_log_size":160767,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1361732,"kb_used_data":455700,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":281753788,"statfs":{"total":289910292480,"available":288515878912,"internally_reserved":0,"allocated":466636800,"data_stored":465842496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,3,7,11,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17739776,"num_objects":-4331,"num_object_clones":0,"num_object_copies":-8662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005716"},"pg_stats":[{"pgid":"2.f","version":"43'17308","reported_seq":17383,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.614763+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:08.614763+0000","last_peered":"2024-09-18T12:03:08.614763+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:08.614763+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"37'7300","ondisk_log_start":"37'7300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9125888,"num_objects":2228,"num_object_clones":0,"num_object_copies":4456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17308,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1371}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17192","reported_seq":17354,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:08.678261+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:08.678261+0000","last_peered":"2024-09-18T12:03:08.678261+0000","last_clean":"2024-09-18T12:03:08.678261+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:08.678261+0000","last_undegraded":"2024-09-18T12:03:08.678261+0000","last_fullsized":"2024-09-18T12:03:08.678261+0000","mapping_epoch":29,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9175040,"num_objects":2240,"num_object_clones":0,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14384,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17290","reported_seq":17480,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.709352+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:08.709352+0000","last_peered":"2024-09-18T12:03:08.709352+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:08.709352+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9285656,"num_objects":2268,"num_object_clones":0,"num_object_copies":4536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2268,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14148,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1268}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17042","reported_seq":17070,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:08.729431+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:08.729431+0000","last_peered":"2024-09-18T12:03:08.729431+0000","last_clean":"2024-09-18T12:03:08.729431+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:08.729431+0000","last_undegraded":"2024-09-18T12:03:08.729431+0000","last_fullsized":"2024-09-18T12:03:08.729431+0000","mapping_epoch":29,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9084928,"num_objects":2218,"num_object_clones":0,"num_object_copies":4436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17042,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17204","reported_seq":17399,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:08.927562+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:08.927562+0000","last_peered":"2024-09-18T12:03:08.927562+0000","last_clean":"2024-09-18T12:03:08.927562+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:08.927562+0000","last_undegraded":"2024-09-18T12:03:08.927562+0000","last_fullsized":"2024-09-18T12:03:08.927562+0000","mapping_epoch":29,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9187328,"num_objects":2243,"num_object_clones":0,"num_object_copies":4486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14369,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17160","reported_seq":17244,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.894439+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:08.894439+0000","last_peered":"2024-09-18T12:03:08.894439+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:08.894439+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9270482,"num_objects":2265,"num_object_clones":0,"num_object_copies":4530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17158,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17174","reported_seq":17265,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.727541+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:08.727541+0000","last_peered":"2024-09-18T12:03:08.727541+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:08.727541+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9004242,"num_objects":2200,"num_object_clones":0,"num_object_copies":4400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17172,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17388","reported_seq":17617,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:08.893047+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:08.893047+0000","last_peered":"2024-09-18T12:03:08.893047+0000","last_clean":"2024-09-18T12:03:08.893047+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:08.893047+0000","last_undegraded":"2024-09-18T12:03:08.893047+0000","last_fullsized":"2024-09-18T12:03:08.893047+0000","mapping_epoch":29,"log_start":"37'7300","ondisk_log_start":"37'7300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9068544,"num_objects":2214,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13705,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17213","reported_seq":24345,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:10.534578+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:10.534578+0000","last_peered":"2024-09-18T12:03:10.534578+0000","last_clean":"2024-09-18T12:03:10.534578+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:10.534578+0000","last_undegraded":"2024-09-18T12:03:10.534578+0000","last_fullsized":"2024-09-18T12:03:10.534578+0000","mapping_epoch":38,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8979666,"num_objects":2194,"num_object_clones":0,"num_object_copies":4388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17211,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17235","reported_seq":19788,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.893234+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:08.893234+0000","last_peered":"2024-09-18T12:03:08.893234+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:08.893234+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9057490,"num_objects":2213,"num_object_clones":0,"num_object_copies":4426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17233,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-86}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17293","reported_seq":17352,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.779013+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:08.779013+0000","last_peered":"2024-09-18T12:03:08.779013+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:08.779013+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8889554,"num_objects":2172,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17291,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1378}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17201","reported_seq":24614,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:08.753035+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:08.753035+0000","last_peered":"2024-09-18T12:03:08.753035+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:08.753035+0000","last_undegraded":"2024-09-18T12:03:08.753035+0000","last_fullsized":"2024-09-18T12:03:08.753035+0000","mapping_epoch":42,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9408512,"num_objects":2297,"num_object_clones":0,"num_object_copies":4594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13681,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17203","reported_seq":17289,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.847723+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:08.847723+0000","last_peered":"2024-09-18T12:03:08.847723+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:08.847723+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9523817,"num_objects":2326,"num_object_clones":0,"num_object_copies":4652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17202,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17359","reported_seq":24509,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:10.571477+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:10.571477+0000","last_peered":"2024-09-18T12:03:10.571477+0000","last_clean":"2024-09-18T12:03:10.571477+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:10.571477+0000","last_undegraded":"2024-09-18T12:03:10.571477+0000","last_fullsized":"2024-09-18T12:03:10.571477+0000","mapping_epoch":40,"log_start":"37'7300","ondisk_log_start":"37'7300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8861499,"num_objects":2166,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17356,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17022","reported_seq":17107,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.835542+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:08.835542+0000","last_peered":"2024-09-18T12:03:08.835542+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:08.835542+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'7000","ondisk_log_start":"37'7000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8914130,"num_objects":2178,"num_object_clones":0,"num_object_copies":4356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17020,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17280","reported_seq":17367,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:08.852394+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:08.852394+0000","last_peered":"2024-09-18T12:03:08.852394+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:08.852394+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9144740,"num_objects":2236,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17277,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":145981516,"num_objects":35658,"num_object_clones":0,"num_object_copies":71316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35658,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259557,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11859,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":464687104,"data_stored":464623296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160764,"ondisk_log_size":160764,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051695,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":492100,"kb_used_data":185912,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93879740,"statfs":{"total":96636764160,"available":96132853760,"internally_reserved":0,"allocated":190373888,"data_stored":190125253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.28399999999999997},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.39800000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":2.282}]}]},{"osd":1,"up_from":35,"seq":150323855458,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403508,"kb_used_data":105632,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93968332,"statfs":{"total":96636764160,"available":96223571968,"internally_reserved":0,"allocated":108167168,"data_stored":107869684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.76600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466124,"kb_used_data":164156,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93905716,"statfs":{"total":96636764160,"available":96159453184,"internally_reserved":0,"allocated":168095744,"data_stored":167847559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,0,2,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.26500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.23699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167493632,"data_stored":167490153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107642880,"data_stored":107590102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":189550592,"data_stored":189543041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:14.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:03:14.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'17308", 'reported_seq': 17383, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:08.614763+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:03:08.614763+0000', 'last_peered': '2024-09-18T12:03:08.614763+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:03:08.614763+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "37'7300", 'ondisk_log_start': "37'7300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9125888, 'num_objects': 2228, 'num_object_clones': 0, 'num_object_copies': 4456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2228, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17308, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1371}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:14.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:03:14.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'17290", 'reported_seq': 17480, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:08.709352+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:03:08.709352+0000', 'last_peered': '2024-09-18T12:03:08.709352+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:03:08.709352+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "37'7200", 'ondisk_log_start': "37'7200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9285656, 'num_objects': 2268, 'num_object_clones': 0, 'num_object_copies': 4536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2268, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14148, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1268}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:14.361 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:03:14.361 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'17160", 'reported_seq': 17244, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:08.894439+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:03:08.894439+0000', 'last_peered': '2024-09-18T12:03:08.894439+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:03:08.894439+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "37'7100", 'ondisk_log_start': "37'7100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9270482, 'num_objects': 2265, 'num_object_clones': 0, 'num_object_copies': 4530, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17158, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:14.361 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:03:14.361 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'17174", 'reported_seq': 17265, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:08.727541+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:03:08.727541+0000', 'last_peered': '2024-09-18T12:03:08.727541+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:03:08.727541+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "37'7100", 'ondisk_log_start': "37'7100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9004242, 'num_objects': 2200, 'num_object_clones': 0, 'num_object_copies': 4400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17172, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1359}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:14.361 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:03:14.361 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'17235", 'reported_seq': 19788, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:08.893234+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:03:08.893234+0000', 'last_peered': '2024-09-18T12:03:08.893234+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:03:08.893234+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "37'7200", 'ondisk_log_start': "37'7200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9057490, 'num_objects': 2213, 'num_object_clones': 0, 'num_object_copies': 4426, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2299, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17233, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1237, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -86}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:14.361 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:03:14.361 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'17293", 'reported_seq': 17352, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:08.779013+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:03:08.779013+0000', 'last_peered': '2024-09-18T12:03:08.779013+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:03:08.779013+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "37'7200", 'ondisk_log_start': "37'7200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8889554, 'num_objects': 2172, 'num_object_clones': 0, 'num_object_copies': 4344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17291, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1378}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:14.361 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:03:14.361 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'17201", 'reported_seq': 24614, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:03:08.753035+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:03:08.753035+0000', 'last_peered': '2024-09-18T12:03:08.753035+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:03:08.753035+0000', 'last_undegraded': '2024-09-18T12:03:08.753035+0000', 'last_fullsized': '2024-09-18T12:03:08.753035+0000', 'mapping_epoch': 42, 'log_start': "37'7200", 'ondisk_log_start': "37'7200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9408512, 'num_objects': 2297, 'num_object_clones': 0, 'num_object_copies': 4594, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2297, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13681, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 2297}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:14.362 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:03:14.362 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'17203", 'reported_seq': 17289, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:08.847723+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:03:08.847723+0000', 'last_peered': '2024-09-18T12:03:08.847723+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:03:08.847723+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "37'7200", 'ondisk_log_start': "37'7200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9523817, 'num_objects': 2326, 'num_object_clones': 0, 'num_object_copies': 4652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17202, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1209}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:14.362 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:03:14.362 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'17022", 'reported_seq': 17107, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:08.835542+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:03:08.835542+0000', 'last_peered': '2024-09-18T12:03:08.835542+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:03:08.835542+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'7000", 'ondisk_log_start': "37'7000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8914130, 'num_objects': 2178, 'num_object_clones': 0, 'num_object_copies': 4356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17020, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1415}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:14.362 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:03:14.362 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'17280", 'reported_seq': 17367, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:08.852394+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:03:08.852394+0000', 'last_peered': '2024-09-18T12:03:08.852394+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:03:08.852394+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "37'7200", 'ondisk_log_start': "37'7200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9144740, 'num_objects': 2236, 'num_object_clones': 0, 'num_object_copies': 4472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17277, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:14.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:03:15.120 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:15.120 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30734/71320 objects degraded (43.093%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":595,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":35660,"data_bytes":145981535,"bytes_used":1394413568,"bytes_avail":288515878912,"bytes_total":289910292480,"degraded_objects":30734,"degraded_total":71320,"degraded_ratio":0.43093101514301746,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":360},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:03:15.168 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:03:15.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:15.852 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:15.865 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:15.894 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-18T12:03:15.242115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":138002527,"num_objects":33712,"num_object_clones":0,"num_object_copies":67424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33712,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":261507,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11936,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160715,"ondisk_log_size":160715,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1358876,"kb_used_data":444532,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281756644,"statfs":{"total":289910292480,"available":288518803456,"internally_reserved":0,"allocated":455200768,"data_stored":454406464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,9,14,12,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25718784,"num_objects":-6279,"num_object_clones":0,"num_object_copies":-12558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6279,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002595"},"pg_stats":[{"pgid":"2.f","version":"43'17411","reported_seq":17482,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:13.896771+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:13.896771+0000","last_peered":"2024-09-18T12:03:13.896771+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:13.896771+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8704000,"num_objects":2125,"num_object_clones":0,"num_object_copies":4250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17411,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1474}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17327","reported_seq":17489,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:13.713117+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:13.713117+0000","last_peered":"2024-09-18T12:03:13.713117+0000","last_clean":"2024-09-18T12:03:13.713117+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:13.713117+0000","last_undegraded":"2024-09-18T12:03:13.713117+0000","last_fullsized":"2024-09-18T12:03:13.713117+0000","mapping_epoch":29,"log_start":"38'7300","ondisk_log_start":"38'7300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8622080,"num_objects":2105,"num_object_clones":0,"num_object_copies":4210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14519,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17430","reported_seq":17620,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:13.766606+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:13.766606+0000","last_peered":"2024-09-18T12:03:13.766606+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:13.766606+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8712216,"num_objects":2128,"num_object_clones":0,"num_object_copies":4256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14288,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1408}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17199","reported_seq":17232,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:13.764239+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:13.764239+0000","last_peered":"2024-09-18T12:03:13.764239+0000","last_clean":"2024-09-18T12:03:13.764239+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:13.764239+0000","last_undegraded":"2024-09-18T12:03:13.764239+0000","last_fullsized":"2024-09-18T12:03:13.764239+0000","mapping_epoch":29,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8441856,"num_objects":2061,"num_object_clones":0,"num_object_copies":4122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17199,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17357","reported_seq":17551,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:13.909776+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:13.909776+0000","last_peered":"2024-09-18T12:03:13.909776+0000","last_clean":"2024-09-18T12:03:13.909776+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:13.909776+0000","last_undegraded":"2024-09-18T12:03:13.909776+0000","last_fullsized":"2024-09-18T12:03:13.909776+0000","mapping_epoch":29,"log_start":"38'7300","ondisk_log_start":"38'7300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8560640,"num_objects":2090,"num_object_clones":0,"num_object_copies":4180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14522,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17297","reported_seq":17382,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:13.903650+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:13.903650+0000","last_peered":"2024-09-18T12:03:13.903650+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:13.903650+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8709330,"num_objects":2128,"num_object_clones":0,"num_object_copies":4256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17295,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17307","reported_seq":17398,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:13.946341+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:13.946341+0000","last_peered":"2024-09-18T12:03:13.946341+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:13.946341+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"38'7300","ondisk_log_start":"38'7300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8459474,"num_objects":2067,"num_object_clones":0,"num_object_copies":4134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17305,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17521","reported_seq":17750,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:13.937673+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:13.937673+0000","last_peered":"2024-09-18T12:03:13.937673+0000","last_clean":"2024-09-18T12:03:13.937673+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:13.937673+0000","last_undegraded":"2024-09-18T12:03:13.937673+0000","last_fullsized":"2024-09-18T12:03:13.937673+0000","mapping_epoch":29,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8523776,"num_objects":2081,"num_object_clones":0,"num_object_copies":4162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13838,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17213","reported_seq":24345,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:10.534578+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:10.534578+0000","last_peered":"2024-09-18T12:03:10.534578+0000","last_clean":"2024-09-18T12:03:10.534578+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:10.534578+0000","last_undegraded":"2024-09-18T12:03:10.534578+0000","last_fullsized":"2024-09-18T12:03:10.534578+0000","mapping_epoch":38,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8979666,"num_objects":2194,"num_object_clones":0,"num_object_copies":4388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17211,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17365","reported_seq":20070,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:13.904395+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:13.904395+0000","last_peered":"2024-09-18T12:03:13.904395+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:13.904395+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"37'7300","ondisk_log_start":"37'7300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8525010,"num_objects":2083,"num_object_clones":0,"num_object_copies":4166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17363,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-140}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17423","reported_seq":17482,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:13.815949+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:13.815949+0000","last_peered":"2024-09-18T12:03:13.815949+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:13.815949+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8357074,"num_objects":2042,"num_object_clones":0,"num_object_copies":4084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17421,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1508}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17338","reported_seq":24751,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:13.791015+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:13.791015+0000","last_peered":"2024-09-18T12:03:13.791015+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:13.791015+0000","last_undegraded":"2024-09-18T12:03:13.791015+0000","last_fullsized":"2024-09-18T12:03:13.791015+0000","mapping_epoch":42,"log_start":"39'7300","ondisk_log_start":"39'7300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8847360,"num_objects":2160,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13818,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17369","reported_seq":17452,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:13.849686+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:13.849686+0000","last_peered":"2024-09-18T12:03:13.849686+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:13.849686+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'7300","ondisk_log_start":"39'7300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8843881,"num_objects":2160,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17368,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17359","reported_seq":24509,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:10.571477+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:10.571477+0000","last_peered":"2024-09-18T12:03:10.571477+0000","last_clean":"2024-09-18T12:03:10.571477+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:10.571477+0000","last_undegraded":"2024-09-18T12:03:10.571477+0000","last_fullsized":"2024-09-18T12:03:10.571477+0000","mapping_epoch":40,"log_start":"37'7300","ondisk_log_start":"37'7300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8861499,"num_objects":2166,"num_object_clones":0,"num_object_copies":4332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17356,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17163","reported_seq":17248,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:13.909899+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:13.909899+0000","last_peered":"2024-09-18T12:03:13.909899+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:13.909899+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'7100","ondisk_log_start":"37'7100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8336594,"num_objects":2037,"num_object_clones":0,"num_object_copies":4074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17161,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1556}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17433","reported_seq":17522,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:13.867510+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:13.867510+0000","last_peered":"2024-09-18T12:03:13.867510+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:13.867510+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8518052,"num_objects":2083,"num_object_clones":0,"num_object_copies":4166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17430,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1440}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":138002508,"num_objects":33710,"num_object_clones":0,"num_object_copies":67420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33710,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":261505,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11935,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":453312512,"data_stored":453248704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160712,"ondisk_log_size":160712,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051696,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491800,"kb_used_data":181456,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93880040,"statfs":{"total":96636764160,"available":96133160960,"internally_reserved":0,"allocated":185810944,"data_stored":185562309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":3.7629999999999999},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.246}]},{"osd":1,"last update":"Wed Sep 18 12:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.657,"15min":1.8700000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.3890000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":3.7109999999999999},{"interface":"front","average":{"1min":2.1499999999999999,"5min":1.3640000000000001,"15min":1.0549999999999999},"min":{"1min":0.28199999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":12.712,"5min":12.712,"15min":12.712},"last":0.39700000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855458,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403508,"kb_used_data":105632,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93968332,"statfs":{"total":96636764160,"available":96223571968,"internally_reserved":0,"allocated":108167168,"data_stored":107869684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,0,4,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.76600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247923,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463568,"kb_used_data":157444,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93908272,"statfs":{"total":96636764160,"available":96162070528,"internally_reserved":0,"allocated":161222656,"data_stored":160974471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,10,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.26800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":5.1440000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":5.1289999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160686080,"data_stored":160682601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107642880,"data_stored":107590102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184983552,"data_stored":184976001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:18.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:19.590 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:19.603 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:19.610 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-18T12:03:19.243023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129056863,"num_objects":31528,"num_object_clones":0,"num_object_copies":63056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31528,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263691,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12018,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160899,"ondisk_log_size":160899,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1341408,"kb_used_data":427072,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281774112,"statfs":{"total":289910292480,"available":288536690688,"internally_reserved":0,"allocated":437321728,"data_stored":436527424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,11,6,13,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25743360,"num_objects":-6285,"num_object_clones":0,"num_object_copies":-12570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6285,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002628"},"pg_stats":[{"pgid":"2.f","version":"43'17552","reported_seq":17627,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.873206+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:18.873206+0000","last_peered":"2024-09-18T12:03:18.873206+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:18.873206+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8126464,"num_objects":1984,"num_object_clones":0,"num_object_copies":3968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17552,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1615}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17479","reported_seq":17641,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:18.834125+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:18.834125+0000","last_peered":"2024-09-18T12:03:18.834125+0000","last_clean":"2024-09-18T12:03:18.834125+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:18.834125+0000","last_undegraded":"2024-09-18T12:03:18.834125+0000","last_fullsized":"2024-09-18T12:03:18.834125+0000","mapping_epoch":29,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7999488,"num_objects":1953,"num_object_clones":0,"num_object_copies":3906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14671,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17569","reported_seq":17758,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.812810+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:18.812810+0000","last_peered":"2024-09-18T12:03:18.812810+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:18.812810+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8142872,"num_objects":1989,"num_object_clones":0,"num_object_copies":3978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1989,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14427,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1547}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17324","reported_seq":17357,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:18.788155+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:18.788155+0000","last_peered":"2024-09-18T12:03:18.788155+0000","last_clean":"2024-09-18T12:03:18.788155+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:18.788155+0000","last_undegraded":"2024-09-18T12:03:18.788155+0000","last_fullsized":"2024-09-18T12:03:18.788155+0000","mapping_epoch":29,"log_start":"39'7300","ondisk_log_start":"39'7300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7929856,"num_objects":1936,"num_object_clones":0,"num_object_copies":3872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17324,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17502","reported_seq":17697,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:18.900597+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:18.900597+0000","last_peered":"2024-09-18T12:03:18.900597+0000","last_clean":"2024-09-18T12:03:18.900597+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:18.900597+0000","last_undegraded":"2024-09-18T12:03:18.900597+0000","last_fullsized":"2024-09-18T12:03:18.900597+0000","mapping_epoch":29,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7966720,"num_objects":1945,"num_object_clones":0,"num_object_copies":3890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14667,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17440","reported_seq":17525,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.899923+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:18.899923+0000","last_peered":"2024-09-18T12:03:18.899923+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:18.899923+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8123602,"num_objects":1985,"num_object_clones":0,"num_object_copies":3970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17438,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1542}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17442","reported_seq":17533,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.867624+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:18.867624+0000","last_peered":"2024-09-18T12:03:18.867624+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:18.867624+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7906514,"num_objects":1932,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17440,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1627}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17640","reported_seq":17869,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:18.858729+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:18.858729+0000","last_peered":"2024-09-18T12:03:18.858729+0000","last_clean":"2024-09-18T12:03:18.858729+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:18.858729+0000","last_undegraded":"2024-09-18T12:03:18.858729+0000","last_fullsized":"2024-09-18T12:03:18.858729+0000","mapping_epoch":29,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8036352,"num_objects":1962,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13957,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17349","reported_seq":24481,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:15.572867+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:15.572867+0000","last_peered":"2024-09-18T12:03:15.572867+0000","last_clean":"2024-09-18T12:03:15.572867+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:15.572867+0000","last_undegraded":"2024-09-18T12:03:15.572867+0000","last_fullsized":"2024-09-18T12:03:15.572867+0000","mapping_epoch":38,"log_start":"39'7300","ondisk_log_start":"39'7300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8422610,"num_objects":2058,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17347,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17491","reported_seq":20360,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.950616+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:18.950616+0000","last_peered":"2024-09-18T12:03:18.950616+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:18.950616+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8008914,"num_objects":1957,"num_object_clones":0,"num_object_copies":3914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17489,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1395,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-184}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17550","reported_seq":17610,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.814020+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:18.814020+0000","last_peered":"2024-09-18T12:03:18.814020+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:18.814020+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7836882,"num_objects":1915,"num_object_clones":0,"num_object_copies":3830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17548,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1635}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17476","reported_seq":24889,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:18.846299+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:18.846299+0000","last_peered":"2024-09-18T12:03:18.846299+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:18.846299+0000","last_undegraded":"2024-09-18T12:03:18.846299+0000","last_fullsized":"2024-09-18T12:03:18.846299+0000","mapping_epoch":42,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8282112,"num_objects":2022,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13956,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2022}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17517","reported_seq":17601,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.921485+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:18.921485+0000","last_peered":"2024-09-18T12:03:18.921485+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:18.921485+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8237673,"num_objects":2012,"num_object_clones":0,"num_object_copies":4024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17516,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1523}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17498","reported_seq":24648,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:15.573033+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:15.573033+0000","last_peered":"2024-09-18T12:03:15.573033+0000","last_clean":"2024-09-18T12:03:15.573033+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:15.573033+0000","last_undegraded":"2024-09-18T12:03:15.573033+0000","last_fullsized":"2024-09-18T12:03:15.573033+0000","mapping_epoch":40,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8292155,"num_objects":2027,"num_object_clones":0,"num_object_copies":4054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17495,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17300","reported_seq":17385,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.913965+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:18.913965+0000","last_peered":"2024-09-18T12:03:18.913965+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:18.913965+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7775442,"num_objects":1900,"num_object_clones":0,"num_object_copies":3800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17298,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17567","reported_seq":17656,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.858610+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:18.858610+0000","last_peered":"2024-09-18T12:03:18.858610+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:18.858610+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7969188,"num_objects":1949,"num_object_clones":0,"num_object_copies":3898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17564,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1574}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":129056844,"num_objects":31526,"num_object_clones":0,"num_object_copies":63052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263689,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12017,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":435425280,"data_stored":435361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160896,"ondisk_log_size":160896,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051697,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474984,"kb_used_data":177120,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93896856,"statfs":{"total":96636764160,"available":96150380544,"internally_reserved":0,"allocated":181370880,"data_stored":181122245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.254},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":1.782}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":1.8260000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":1.8540000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855459,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401268,"kb_used_data":99240,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93970572,"statfs":{"total":96636764160,"available":96225865728,"internally_reserved":0,"allocated":101621760,"data_stored":101324276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,6,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.218}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.26800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465156,"kb_used_data":150712,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93906684,"statfs":{"total":96636764160,"available":96160444416,"internally_reserved":0,"allocated":154329088,"data_stored":154080903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":321973766},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,7,2,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.253},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.33600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153890816,"data_stored":153887337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101007360,"data_stored":100954582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180527104,"data_stored":180519553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:19.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:20.170 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:20.182 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:20.218 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-18T12:03:19.243023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129056863,"num_objects":31528,"num_object_clones":0,"num_object_copies":63056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31528,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263691,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12018,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160899,"ondisk_log_size":160899,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1341408,"kb_used_data":427072,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281774112,"statfs":{"total":289910292480,"available":288536690688,"internally_reserved":0,"allocated":437321728,"data_stored":436527424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,11,6,13,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25743360,"num_objects":-6285,"num_object_clones":0,"num_object_copies":-12570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6285,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002628"},"pg_stats":[{"pgid":"2.f","version":"43'17552","reported_seq":17627,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.873206+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:18.873206+0000","last_peered":"2024-09-18T12:03:18.873206+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:18.873206+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8126464,"num_objects":1984,"num_object_clones":0,"num_object_copies":3968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17552,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1615}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17479","reported_seq":17641,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:18.834125+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:18.834125+0000","last_peered":"2024-09-18T12:03:18.834125+0000","last_clean":"2024-09-18T12:03:18.834125+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:18.834125+0000","last_undegraded":"2024-09-18T12:03:18.834125+0000","last_fullsized":"2024-09-18T12:03:18.834125+0000","mapping_epoch":29,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7999488,"num_objects":1953,"num_object_clones":0,"num_object_copies":3906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14671,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17569","reported_seq":17758,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.812810+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:18.812810+0000","last_peered":"2024-09-18T12:03:18.812810+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:18.812810+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8142872,"num_objects":1989,"num_object_clones":0,"num_object_copies":3978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1989,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14427,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1547}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17324","reported_seq":17357,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:18.788155+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:18.788155+0000","last_peered":"2024-09-18T12:03:18.788155+0000","last_clean":"2024-09-18T12:03:18.788155+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:18.788155+0000","last_undegraded":"2024-09-18T12:03:18.788155+0000","last_fullsized":"2024-09-18T12:03:18.788155+0000","mapping_epoch":29,"log_start":"39'7300","ondisk_log_start":"39'7300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7929856,"num_objects":1936,"num_object_clones":0,"num_object_copies":3872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17324,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17502","reported_seq":17697,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:18.900597+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:18.900597+0000","last_peered":"2024-09-18T12:03:18.900597+0000","last_clean":"2024-09-18T12:03:18.900597+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:18.900597+0000","last_undegraded":"2024-09-18T12:03:18.900597+0000","last_fullsized":"2024-09-18T12:03:18.900597+0000","mapping_epoch":29,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7966720,"num_objects":1945,"num_object_clones":0,"num_object_copies":3890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14667,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17440","reported_seq":17525,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.899923+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:18.899923+0000","last_peered":"2024-09-18T12:03:18.899923+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:18.899923+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8123602,"num_objects":1985,"num_object_clones":0,"num_object_copies":3970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17438,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1542}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17442","reported_seq":17533,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.867624+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:18.867624+0000","last_peered":"2024-09-18T12:03:18.867624+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:18.867624+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7906514,"num_objects":1932,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17440,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1627}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17640","reported_seq":17869,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:18.858729+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:18.858729+0000","last_peered":"2024-09-18T12:03:18.858729+0000","last_clean":"2024-09-18T12:03:18.858729+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:18.858729+0000","last_undegraded":"2024-09-18T12:03:18.858729+0000","last_fullsized":"2024-09-18T12:03:18.858729+0000","mapping_epoch":29,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8036352,"num_objects":1962,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13957,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17349","reported_seq":24481,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:15.572867+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:15.572867+0000","last_peered":"2024-09-18T12:03:15.572867+0000","last_clean":"2024-09-18T12:03:15.572867+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:15.572867+0000","last_undegraded":"2024-09-18T12:03:15.572867+0000","last_fullsized":"2024-09-18T12:03:15.572867+0000","mapping_epoch":38,"log_start":"39'7300","ondisk_log_start":"39'7300","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8422610,"num_objects":2058,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17347,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17491","reported_seq":20360,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.950616+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:18.950616+0000","last_peered":"2024-09-18T12:03:18.950616+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:18.950616+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8008914,"num_objects":1957,"num_object_clones":0,"num_object_copies":3914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17489,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1395,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-184}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17550","reported_seq":17610,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.814020+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:18.814020+0000","last_peered":"2024-09-18T12:03:18.814020+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:18.814020+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7836882,"num_objects":1915,"num_object_clones":0,"num_object_copies":3830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17548,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1635}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17476","reported_seq":24889,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:18.846299+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:18.846299+0000","last_peered":"2024-09-18T12:03:18.846299+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:18.846299+0000","last_undegraded":"2024-09-18T12:03:18.846299+0000","last_fullsized":"2024-09-18T12:03:18.846299+0000","mapping_epoch":42,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8282112,"num_objects":2022,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13956,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2022}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17517","reported_seq":17601,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.921485+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:18.921485+0000","last_peered":"2024-09-18T12:03:18.921485+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:18.921485+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8237673,"num_objects":2012,"num_object_clones":0,"num_object_copies":4024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17516,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1523}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17498","reported_seq":24648,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:15.573033+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:15.573033+0000","last_peered":"2024-09-18T12:03:15.573033+0000","last_clean":"2024-09-18T12:03:15.573033+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:15.573033+0000","last_undegraded":"2024-09-18T12:03:15.573033+0000","last_fullsized":"2024-09-18T12:03:15.573033+0000","mapping_epoch":40,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8292155,"num_objects":2027,"num_object_clones":0,"num_object_copies":4054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17495,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17300","reported_seq":17385,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.913965+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:18.913965+0000","last_peered":"2024-09-18T12:03:18.913965+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:18.913965+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7775442,"num_objects":1900,"num_object_clones":0,"num_object_copies":3800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17298,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17567","reported_seq":17656,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.858610+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:18.858610+0000","last_peered":"2024-09-18T12:03:18.858610+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:18.858610+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7969188,"num_objects":1949,"num_object_clones":0,"num_object_copies":3898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17564,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1574}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":129056844,"num_objects":31526,"num_object_clones":0,"num_object_copies":63052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263689,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12017,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":435425280,"data_stored":435361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160896,"ondisk_log_size":160896,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051697,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474984,"kb_used_data":177120,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93896856,"statfs":{"total":96636764160,"available":96150380544,"internally_reserved":0,"allocated":181370880,"data_stored":181122245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.254},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":1.782}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":1.8260000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":1.8540000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855459,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401268,"kb_used_data":99240,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93970572,"statfs":{"total":96636764160,"available":96225865728,"internally_reserved":0,"allocated":101621760,"data_stored":101324276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,6,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.218}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":0.26800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465156,"kb_used_data":150712,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93906684,"statfs":{"total":96636764160,"available":96160444416,"internally_reserved":0,"allocated":154329088,"data_stored":154080903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":321973766},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,7,2,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.253},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.33600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153890816,"data_stored":153887337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101007360,"data_stored":100954582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180527104,"data_stored":180519553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:20.220 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:03:20.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'17552", 'reported_seq': 17627, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:18.873206+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:03:18.873206+0000', 'last_peered': '2024-09-18T12:03:18.873206+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:03:18.873206+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'7500", 'ondisk_log_start': "39'7500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8126464, 'num_objects': 1984, 'num_object_clones': 0, 'num_object_copies': 3968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17552, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1615}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:20.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:03:20.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'17569", 'reported_seq': 17758, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:18.812810+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:03:18.812810+0000', 'last_peered': '2024-09-18T12:03:18.812810+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:03:18.812810+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'7500", 'ondisk_log_start': "39'7500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8142872, 'num_objects': 1989, 'num_object_clones': 0, 'num_object_copies': 3978, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1989, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14427, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1547}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:20.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:03:20.221 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'17440", 'reported_seq': 17525, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:18.899923+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:03:18.899923+0000', 'last_peered': '2024-09-18T12:03:18.899923+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:03:18.899923+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'7400", 'ondisk_log_start': "39'7400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8123602, 'num_objects': 1985, 'num_object_clones': 0, 'num_object_copies': 3970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17438, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1542}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:20.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:03:20.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'17442", 'reported_seq': 17533, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:18.867624+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:03:18.867624+0000', 'last_peered': '2024-09-18T12:03:18.867624+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:03:18.867624+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'7400", 'ondisk_log_start': "39'7400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7906514, 'num_objects': 1932, 'num_object_clones': 0, 'num_object_copies': 3864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17440, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1627}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:20.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:03:20.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'17491", 'reported_seq': 20360, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:18.950616+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:03:18.950616+0000', 'last_peered': '2024-09-18T12:03:18.950616+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:03:18.950616+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'7400", 'ondisk_log_start': "39'7400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8008914, 'num_objects': 1957, 'num_object_clones': 0, 'num_object_copies': 3914, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2141, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17489, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1395, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -184}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:20.222 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:03:20.223 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'17550", 'reported_seq': 17610, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:18.814020+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:03:18.814020+0000', 'last_peered': '2024-09-18T12:03:18.814020+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:03:18.814020+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'7500", 'ondisk_log_start': "39'7500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7836882, 'num_objects': 1915, 'num_object_clones': 0, 'num_object_copies': 3830, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1915, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17548, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1635}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:20.223 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:03:20.223 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'17476", 'reported_seq': 24889, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:03:18.846299+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:03:18.846299+0000', 'last_peered': '2024-09-18T12:03:18.846299+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:03:18.846299+0000', 'last_undegraded': '2024-09-18T12:03:18.846299+0000', 'last_fullsized': '2024-09-18T12:03:18.846299+0000', 'mapping_epoch': 42, 'log_start': "39'7400", 'ondisk_log_start': "39'7400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8282112, 'num_objects': 2022, 'num_object_clones': 0, 'num_object_copies': 4044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2022, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13956, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 2022}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:20.223 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:03:20.223 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'17517", 'reported_seq': 17601, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:18.921485+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:03:18.921485+0000', 'last_peered': '2024-09-18T12:03:18.921485+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:03:18.921485+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'7500", 'ondisk_log_start': "39'7500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8237673, 'num_objects': 2012, 'num_object_clones': 0, 'num_object_copies': 4024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17516, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1523}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:20.224 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:03:20.224 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'17300", 'reported_seq': 17385, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:18.913965+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:03:18.913965+0000', 'last_peered': '2024-09-18T12:03:18.913965+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:03:18.913965+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "37'7200", 'ondisk_log_start': "37'7200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7775442, 'num_objects': 1900, 'num_object_clones': 0, 'num_object_copies': 3800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17298, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1693}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:20.224 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:03:20.224 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'17567", 'reported_seq': 17656, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:18.858610+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:03:18.858610+0000', 'last_peered': '2024-09-18T12:03:18.858610+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:03:18.858610+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'7500", 'ondisk_log_start': "39'7500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7969188, 'num_objects': 1949, 'num_object_clones': 0, 'num_object_copies': 3898, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1949, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17564, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1574}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:20.225 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:03:21.149 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:21.149 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30576/63056 objects degraded (48.490%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":601,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":31528,"data_bytes":129056863,"bytes_used":1373601792,"bytes_avail":288536690688,"bytes_total":289910292480,"degraded_objects":30576,"degraded_total":63056,"degraded_ratio":0.48490230905861459,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":238,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":523},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:03:21.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:03:21.189 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:21.916 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:21.934 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:21.968 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-18T12:03:21.243475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127901791,"num_objects":31246,"num_object_clones":0,"num_object_copies":62492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263973,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12018,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160881,"ondisk_log_size":160881,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1342924,"kb_used_data":420268,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281772596,"statfs":{"total":289910292480,"available":288535138304,"internally_reserved":0,"allocated":430354432,"data_stored":429560128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,11,4,13,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19148800,"num_objects":-4675,"num_object_clones":0,"num_object_copies":-9350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4675,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002646"},"pg_stats":[{"pgid":"2.f","version":"43'17552","reported_seq":17627,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.873206+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:18.873206+0000","last_peered":"2024-09-18T12:03:18.873206+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:18.873206+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8126464,"num_objects":1984,"num_object_clones":0,"num_object_copies":3968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17552,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1615}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17479","reported_seq":17641,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:18.834125+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:18.834125+0000","last_peered":"2024-09-18T12:03:18.834125+0000","last_clean":"2024-09-18T12:03:18.834125+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:18.834125+0000","last_undegraded":"2024-09-18T12:03:18.834125+0000","last_fullsized":"2024-09-18T12:03:18.834125+0000","mapping_epoch":29,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7999488,"num_objects":1953,"num_object_clones":0,"num_object_copies":3906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14671,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17569","reported_seq":17758,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.812810+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:18.812810+0000","last_peered":"2024-09-18T12:03:18.812810+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:18.812810+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8142872,"num_objects":1989,"num_object_clones":0,"num_object_copies":3978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1989,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14427,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1547}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17324","reported_seq":17357,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:18.788155+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:18.788155+0000","last_peered":"2024-09-18T12:03:18.788155+0000","last_clean":"2024-09-18T12:03:18.788155+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:18.788155+0000","last_undegraded":"2024-09-18T12:03:18.788155+0000","last_fullsized":"2024-09-18T12:03:18.788155+0000","mapping_epoch":29,"log_start":"39'7300","ondisk_log_start":"39'7300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7929856,"num_objects":1936,"num_object_clones":0,"num_object_copies":3872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17324,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17502","reported_seq":17697,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:18.900597+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:18.900597+0000","last_peered":"2024-09-18T12:03:18.900597+0000","last_clean":"2024-09-18T12:03:18.900597+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:18.900597+0000","last_undegraded":"2024-09-18T12:03:18.900597+0000","last_fullsized":"2024-09-18T12:03:18.900597+0000","mapping_epoch":29,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7966720,"num_objects":1945,"num_object_clones":0,"num_object_copies":3890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14667,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17440","reported_seq":17525,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.899923+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:18.899923+0000","last_peered":"2024-09-18T12:03:18.899923+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:18.899923+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8123602,"num_objects":1985,"num_object_clones":0,"num_object_copies":3970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17438,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1542}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17442","reported_seq":17533,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.867624+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:18.867624+0000","last_peered":"2024-09-18T12:03:18.867624+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:18.867624+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7906514,"num_objects":1932,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17440,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1627}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17640","reported_seq":17869,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:18.858729+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:18.858729+0000","last_peered":"2024-09-18T12:03:18.858729+0000","last_clean":"2024-09-18T12:03:18.858729+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:18.858729+0000","last_undegraded":"2024-09-18T12:03:18.858729+0000","last_fullsized":"2024-09-18T12:03:18.858729+0000","mapping_epoch":29,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8036352,"num_objects":1962,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13957,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17487","reported_seq":24619,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:20.650278+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:20.650278+0000","last_peered":"2024-09-18T12:03:20.650278+0000","last_clean":"2024-09-18T12:03:20.650278+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:20.650278+0000","last_undegraded":"2024-09-18T12:03:20.650278+0000","last_fullsized":"2024-09-18T12:03:20.650278+0000","mapping_epoch":38,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7857362,"num_objects":1920,"num_object_clones":0,"num_object_copies":3840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17485,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17491","reported_seq":20360,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.950616+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:18.950616+0000","last_peered":"2024-09-18T12:03:18.950616+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:18.950616+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8008914,"num_objects":1957,"num_object_clones":0,"num_object_copies":3914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17489,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1395,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-184}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17550","reported_seq":17610,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.814020+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:18.814020+0000","last_peered":"2024-09-18T12:03:18.814020+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:18.814020+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7836882,"num_objects":1915,"num_object_clones":0,"num_object_copies":3830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17548,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1635}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17476","reported_seq":24889,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:18.846299+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:18.846299+0000","last_peered":"2024-09-18T12:03:18.846299+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:18.846299+0000","last_undegraded":"2024-09-18T12:03:18.846299+0000","last_fullsized":"2024-09-18T12:03:18.846299+0000","mapping_epoch":42,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8282112,"num_objects":2022,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13956,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":2022}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17517","reported_seq":17601,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.921485+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:18.921485+0000","last_peered":"2024-09-18T12:03:18.921485+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:18.921485+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8237673,"num_objects":2012,"num_object_clones":0,"num_object_copies":4024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17516,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1523}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17642","reported_seq":24792,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:20.534622+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:20.534622+0000","last_peered":"2024-09-18T12:03:20.534622+0000","last_clean":"2024-09-18T12:03:20.534622+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:20.534622+0000","last_undegraded":"2024-09-18T12:03:20.534622+0000","last_fullsized":"2024-09-18T12:03:20.534622+0000","mapping_epoch":40,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7702331,"num_objects":1883,"num_object_clones":0,"num_object_copies":3766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17639,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17300","reported_seq":17385,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.913965+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:18.913965+0000","last_peered":"2024-09-18T12:03:18.913965+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:18.913965+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"37'7200","ondisk_log_start":"37'7200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7775442,"num_objects":1900,"num_object_clones":0,"num_object_copies":3800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17298,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17567","reported_seq":17656,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:18.858610+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:18.858610+0000","last_peered":"2024-09-18T12:03:18.858610+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:18.858610+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7969188,"num_objects":1949,"num_object_clones":0,"num_object_copies":3898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17564,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1574}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":127901772,"num_objects":31244,"num_object_clones":0,"num_object_copies":62488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31244,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263971,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12017,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":428584960,"data_stored":428521152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160878,"ondisk_log_size":160878,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051697,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474984,"kb_used_data":177120,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93896856,"statfs":{"total":96636764160,"available":96150380544,"internally_reserved":0,"allocated":181370880,"data_stored":181122245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.254},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":1.782}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":1.8260000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":1.8540000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855460,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402784,"kb_used_data":92436,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93969056,"statfs":{"total":96636764160,"available":96224313344,"internally_reserved":0,"allocated":94654464,"data_stored":94356980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":5.2869999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.221}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":5.1289999999999996},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":5.0030000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465156,"kb_used_data":150712,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93906684,"statfs":{"total":96636764160,"available":96160444416,"internally_reserved":0,"allocated":154329088,"data_stored":154080903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":321973766},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,7,2,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.3540000000000001,"15min":1.1020000000000001},"min":{"1min":0.186,"5min":0.161,"15min":0.161},"max":{"1min":11.521000000000001,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.91300000000000003,"15min":1.0580000000000001},"min":{"1min":0.20300000000000001,"5min":0.189,"15min":0.189},"max":{"1min":5.5590000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.253},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.33600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153890816,"data_stored":153887337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94167040,"data_stored":94114262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180527104,"data_stored":180519553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:24.970 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:25.798 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:25.815 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:25.833 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-18T12:03:25.244312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120037471,"num_objects":29326,"num_object_clones":0,"num_object_copies":58652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":265893,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12090,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1323496,"kb_used_data":409164,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281792024,"statfs":{"total":289910292480,"available":288555032576,"internally_reserved":0,"allocated":418983936,"data_stored":418189632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,6,2,11,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25944064,"num_objects":-6334,"num_object_clones":0,"num_object_copies":-12668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6334,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002536"},"pg_stats":[{"pgid":"2.f","version":"43'17686","reported_seq":17761,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.964169+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:23.964169+0000","last_peered":"2024-09-18T12:03:23.964169+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:23.964169+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7577600,"num_objects":1850,"num_object_clones":0,"num_object_copies":3700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17686,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1749}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17617","reported_seq":17779,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:23.825039+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:23.825039+0000","last_peered":"2024-09-18T12:03:23.825039+0000","last_clean":"2024-09-18T12:03:23.825039+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:23.825039+0000","last_undegraded":"2024-09-18T12:03:23.825039+0000","last_fullsized":"2024-09-18T12:03:23.825039+0000","mapping_epoch":29,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7434240,"num_objects":1815,"num_object_clones":0,"num_object_copies":3630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14809,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17720","reported_seq":17910,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.791928+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:23.791928+0000","last_peered":"2024-09-18T12:03:23.791928+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:23.791928+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7524376,"num_objects":1838,"num_object_clones":0,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14578,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1698}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17458","reported_seq":17491,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:23.694057+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:23.694057+0000","last_peered":"2024-09-18T12:03:23.694057+0000","last_clean":"2024-09-18T12:03:23.694057+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:23.694057+0000","last_undegraded":"2024-09-18T12:03:23.694057+0000","last_fullsized":"2024-09-18T12:03:23.694057+0000","mapping_epoch":29,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7380992,"num_objects":1802,"num_object_clones":0,"num_object_copies":3604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17458,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17646","reported_seq":17841,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:23.924633+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:23.924633+0000","last_peered":"2024-09-18T12:03:23.924633+0000","last_clean":"2024-09-18T12:03:23.924633+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:23.924633+0000","last_undegraded":"2024-09-18T12:03:23.924633+0000","last_fullsized":"2024-09-18T12:03:23.924633+0000","mapping_epoch":29,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7376896,"num_objects":1801,"num_object_clones":0,"num_object_copies":3602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14811,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17583","reported_seq":17665,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.966801+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:23.966801+0000","last_peered":"2024-09-18T12:03:23.966801+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:23.966801+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7537874,"num_objects":1842,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17581,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1685}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17576","reported_seq":17667,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.911536+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:23.911536+0000","last_peered":"2024-09-18T12:03:23.911536+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:23.911536+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7357650,"num_objects":1798,"num_object_clones":0,"num_object_copies":3596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17574,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17806","reported_seq":18035,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:23.925625+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:23.925625+0000","last_peered":"2024-09-18T12:03:23.925625+0000","last_clean":"2024-09-18T12:03:23.925625+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:23.925625+0000","last_undegraded":"2024-09-18T12:03:23.925625+0000","last_fullsized":"2024-09-18T12:03:23.925625+0000","mapping_epoch":29,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7356416,"num_objects":1796,"num_object_clones":0,"num_object_copies":3592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14123,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17487","reported_seq":24619,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:20.650278+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:20.650278+0000","last_peered":"2024-09-18T12:03:20.650278+0000","last_clean":"2024-09-18T12:03:20.650278+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:20.650278+0000","last_undegraded":"2024-09-18T12:03:20.650278+0000","last_fullsized":"2024-09-18T12:03:20.650278+0000","mapping_epoch":38,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7857362,"num_objects":1920,"num_object_clones":0,"num_object_copies":3840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17485,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17627","reported_seq":20639,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.965465+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:23.965465+0000","last_peered":"2024-09-18T12:03:23.965465+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:23.965465+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7451858,"num_objects":1821,"num_object_clones":0,"num_object_copies":3642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17625,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-248}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17666","reported_seq":17726,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.790872+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:23.790872+0000","last_peered":"2024-09-18T12:03:23.790872+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:23.790872+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7361746,"num_objects":1799,"num_object_clones":0,"num_object_copies":3598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17664,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1751}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17612","reported_seq":25025,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:23.825493+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:23.825493+0000","last_peered":"2024-09-18T12:03:23.825493+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:23.825493+0000","last_undegraded":"2024-09-18T12:03:23.825493+0000","last_fullsized":"2024-09-18T12:03:23.825493+0000","mapping_epoch":42,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7725056,"num_objects":1886,"num_object_clones":0,"num_object_copies":3772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14092,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1886}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17647","reported_seq":17733,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.930914+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:23.930914+0000","last_peered":"2024-09-18T12:03:23.930914+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:23.930914+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7705193,"num_objects":1882,"num_object_clones":0,"num_object_copies":3764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17646,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17642","reported_seq":24792,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:20.534622+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:20.534622+0000","last_peered":"2024-09-18T12:03:20.534622+0000","last_clean":"2024-09-18T12:03:20.534622+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:20.534622+0000","last_undegraded":"2024-09-18T12:03:20.534622+0000","last_fullsized":"2024-09-18T12:03:20.534622+0000","mapping_epoch":40,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7702331,"num_objects":1883,"num_object_clones":0,"num_object_copies":3766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17639,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17431","reported_seq":17516,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.835633+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:23.835633+0000","last_peered":"2024-09-18T12:03:23.835633+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:23.835633+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7238866,"num_objects":1769,"num_object_clones":0,"num_object_copies":3538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17429,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17694","reported_seq":17783,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.941863+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:23.941863+0000","last_peered":"2024-09-18T12:03:23.941863+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:23.941863+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7448996,"num_objects":1822,"num_object_clones":0,"num_object_copies":3644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17691,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":120037452,"num_objects":29324,"num_object_clones":0,"num_object_copies":58648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":265891,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12089,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417271808,"data_stored":417208000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160798,"ondisk_log_size":160798,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051698,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474652,"kb_used_data":172628,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93897188,"statfs":{"total":96636764160,"available":96150720512,"internally_reserved":0,"allocated":176771072,"data_stored":176522437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.68200000000000005},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":1.3340000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855460,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402784,"kb_used_data":92436,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93969056,"statfs":{"total":96636764160,"available":96224313344,"internally_reserved":0,"allocated":94654464,"data_stored":94356980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":5.2869999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.221}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":5.1289999999999996},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":5.0030000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446060,"kb_used_data":144100,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93925780,"statfs":{"total":96636764160,"available":96179998720,"internally_reserved":0,"allocated":147558400,"data_stored":147310215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20999999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147128320,"data_stored":147124841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94167040,"data_stored":94114262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":175976448,"data_stored":175968897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:25.834 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:26.584 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:26.598 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-18T12:03:25.244312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120037471,"num_objects":29326,"num_object_clones":0,"num_object_copies":58652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":265893,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12090,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1323496,"kb_used_data":409164,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281792024,"statfs":{"total":289910292480,"available":288555032576,"internally_reserved":0,"allocated":418983936,"data_stored":418189632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,6,2,11,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25944064,"num_objects":-6334,"num_object_clones":0,"num_object_copies":-12668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6334,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002536"},"pg_stats":[{"pgid":"2.f","version":"43'17686","reported_seq":17761,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.964169+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:23.964169+0000","last_peered":"2024-09-18T12:03:23.964169+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:23.964169+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7577600,"num_objects":1850,"num_object_clones":0,"num_object_copies":3700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17686,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1749}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17617","reported_seq":17779,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:23.825039+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:23.825039+0000","last_peered":"2024-09-18T12:03:23.825039+0000","last_clean":"2024-09-18T12:03:23.825039+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:23.825039+0000","last_undegraded":"2024-09-18T12:03:23.825039+0000","last_fullsized":"2024-09-18T12:03:23.825039+0000","mapping_epoch":29,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7434240,"num_objects":1815,"num_object_clones":0,"num_object_copies":3630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14809,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17720","reported_seq":17910,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.791928+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:23.791928+0000","last_peered":"2024-09-18T12:03:23.791928+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:23.791928+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7524376,"num_objects":1838,"num_object_clones":0,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14578,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1698}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17458","reported_seq":17491,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:23.694057+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:23.694057+0000","last_peered":"2024-09-18T12:03:23.694057+0000","last_clean":"2024-09-18T12:03:23.694057+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:23.694057+0000","last_undegraded":"2024-09-18T12:03:23.694057+0000","last_fullsized":"2024-09-18T12:03:23.694057+0000","mapping_epoch":29,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7380992,"num_objects":1802,"num_object_clones":0,"num_object_copies":3604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17458,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17646","reported_seq":17841,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:23.924633+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:23.924633+0000","last_peered":"2024-09-18T12:03:23.924633+0000","last_clean":"2024-09-18T12:03:23.924633+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:23.924633+0000","last_undegraded":"2024-09-18T12:03:23.924633+0000","last_fullsized":"2024-09-18T12:03:23.924633+0000","mapping_epoch":29,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7376896,"num_objects":1801,"num_object_clones":0,"num_object_copies":3602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14811,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17583","reported_seq":17665,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.966801+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:23.966801+0000","last_peered":"2024-09-18T12:03:23.966801+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:23.966801+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7537874,"num_objects":1842,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17581,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1685}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17576","reported_seq":17667,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.911536+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:23.911536+0000","last_peered":"2024-09-18T12:03:23.911536+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:23.911536+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7357650,"num_objects":1798,"num_object_clones":0,"num_object_copies":3596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17574,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17806","reported_seq":18035,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:23.925625+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:23.925625+0000","last_peered":"2024-09-18T12:03:23.925625+0000","last_clean":"2024-09-18T12:03:23.925625+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:23.925625+0000","last_undegraded":"2024-09-18T12:03:23.925625+0000","last_fullsized":"2024-09-18T12:03:23.925625+0000","mapping_epoch":29,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7356416,"num_objects":1796,"num_object_clones":0,"num_object_copies":3592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14123,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17487","reported_seq":24619,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:20.650278+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:20.650278+0000","last_peered":"2024-09-18T12:03:20.650278+0000","last_clean":"2024-09-18T12:03:20.650278+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:20.650278+0000","last_undegraded":"2024-09-18T12:03:20.650278+0000","last_fullsized":"2024-09-18T12:03:20.650278+0000","mapping_epoch":38,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7857362,"num_objects":1920,"num_object_clones":0,"num_object_copies":3840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17485,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17627","reported_seq":20639,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.965465+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:23.965465+0000","last_peered":"2024-09-18T12:03:23.965465+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:23.965465+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7451858,"num_objects":1821,"num_object_clones":0,"num_object_copies":3642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17625,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-248}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17666","reported_seq":17726,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.790872+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:23.790872+0000","last_peered":"2024-09-18T12:03:23.790872+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:23.790872+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7361746,"num_objects":1799,"num_object_clones":0,"num_object_copies":3598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17664,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1751}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17612","reported_seq":25025,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:23.825493+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:23.825493+0000","last_peered":"2024-09-18T12:03:23.825493+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:23.825493+0000","last_undegraded":"2024-09-18T12:03:23.825493+0000","last_fullsized":"2024-09-18T12:03:23.825493+0000","mapping_epoch":42,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7725056,"num_objects":1886,"num_object_clones":0,"num_object_copies":3772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14092,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1886}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17647","reported_seq":17733,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.930914+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:23.930914+0000","last_peered":"2024-09-18T12:03:23.930914+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:23.930914+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7705193,"num_objects":1882,"num_object_clones":0,"num_object_copies":3764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17646,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17642","reported_seq":24792,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:20.534622+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:20.534622+0000","last_peered":"2024-09-18T12:03:20.534622+0000","last_clean":"2024-09-18T12:03:20.534622+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:20.534622+0000","last_undegraded":"2024-09-18T12:03:20.534622+0000","last_fullsized":"2024-09-18T12:03:20.534622+0000","mapping_epoch":40,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7702331,"num_objects":1883,"num_object_clones":0,"num_object_copies":3766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17639,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17431","reported_seq":17516,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.835633+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:23.835633+0000","last_peered":"2024-09-18T12:03:23.835633+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:23.835633+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7238866,"num_objects":1769,"num_object_clones":0,"num_object_copies":3538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17429,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17694","reported_seq":17783,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.941863+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:23.941863+0000","last_peered":"2024-09-18T12:03:23.941863+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:23.941863+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7448996,"num_objects":1822,"num_object_clones":0,"num_object_copies":3644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17691,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":120037452,"num_objects":29324,"num_object_clones":0,"num_object_copies":58648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":265891,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12089,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417271808,"data_stored":417208000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160798,"ondisk_log_size":160798,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051698,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474652,"kb_used_data":172628,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93897188,"statfs":{"total":96636764160,"available":96150720512,"internally_reserved":0,"allocated":176771072,"data_stored":176522437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.68200000000000005},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":1.3340000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855460,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402784,"kb_used_data":92436,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93969056,"statfs":{"total":96636764160,"available":96224313344,"internally_reserved":0,"allocated":94654464,"data_stored":94356980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.40100000000000002,"15min":0.435},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.921,"5min":5.3730000000000002,"15min":5.3730000000000002},"last":5.2869999999999999},{"interface":"front","average":{"1min":1.347,"5min":1.7789999999999999,"15min":1.226},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.5739999999999998,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.221}]},{"osd":2,"last update":"Wed Sep 18 12:02:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.7729999999999999,"15min":1.2509999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.737,"5min":13.737,"15min":13.737},"last":5.1289999999999996},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.008,"15min":0.76400000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.4730000000000008,"5min":11.996,"15min":11.996},"last":5.0030000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446060,"kb_used_data":144100,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93925780,"statfs":{"total":96636764160,"available":96179998720,"internally_reserved":0,"allocated":147558400,"data_stored":147310215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20999999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147128320,"data_stored":147124841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94167040,"data_stored":94114262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":175976448,"data_stored":175968897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:26.599 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:03:26.599 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'17686", 'reported_seq': 17761, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:23.964169+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:03:23.964169+0000', 'last_peered': '2024-09-18T12:03:23.964169+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:03:23.964169+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'7600", 'ondisk_log_start': "39'7600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7577600, 'num_objects': 1850, 'num_object_clones': 0, 'num_object_copies': 3700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17686, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1749}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:26.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:03:26.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'17720", 'reported_seq': 17910, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:23.791928+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:03:23.791928+0000', 'last_peered': '2024-09-18T12:03:23.791928+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:03:23.791928+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'7700", 'ondisk_log_start': "39'7700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7524376, 'num_objects': 1838, 'num_object_clones': 0, 'num_object_copies': 3676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1838, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14578, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1698}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:26.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:03:26.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'17583", 'reported_seq': 17665, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:23.966801+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:03:23.966801+0000', 'last_peered': '2024-09-18T12:03:23.966801+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:03:23.966801+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'7500", 'ondisk_log_start': "39'7500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7537874, 'num_objects': 1842, 'num_object_clones': 0, 'num_object_copies': 3684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17581, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1685}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:26.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:03:26.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'17576", 'reported_seq': 17667, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:23.911536+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:03:23.911536+0000', 'last_peered': '2024-09-18T12:03:23.911536+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:03:23.911536+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'7500", 'ondisk_log_start': "39'7500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7357650, 'num_objects': 1798, 'num_object_clones': 0, 'num_object_copies': 3596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17574, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1761}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:26.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:03:26.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'17627", 'reported_seq': 20639, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:23.965465+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:03:23.965465+0000', 'last_peered': '2024-09-18T12:03:23.965465+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:03:23.965465+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'7600", 'ondisk_log_start': "39'7600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7451858, 'num_objects': 1821, 'num_object_clones': 0, 'num_object_copies': 3642, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2069, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17625, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1467, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -248}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:26.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:03:26.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'17666", 'reported_seq': 17726, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:23.790872+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:03:23.790872+0000', 'last_peered': '2024-09-18T12:03:23.790872+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:03:23.790872+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'7600", 'ondisk_log_start': "39'7600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7361746, 'num_objects': 1799, 'num_object_clones': 0, 'num_object_copies': 3598, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17664, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1751}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:26.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:03:26.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'17612", 'reported_seq': 25025, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:03:23.825493+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:03:23.825493+0000', 'last_peered': '2024-09-18T12:03:23.825493+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:03:23.825493+0000', 'last_undegraded': '2024-09-18T12:03:23.825493+0000', 'last_fullsized': '2024-09-18T12:03:23.825493+0000', 'mapping_epoch': 42, 'log_start': "39'7600", 'ondisk_log_start': "39'7600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7725056, 'num_objects': 1886, 'num_object_clones': 0, 'num_object_copies': 3772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14092, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1886}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:26.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:03:26.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'17647", 'reported_seq': 17733, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:23.930914+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:03:23.930914+0000', 'last_peered': '2024-09-18T12:03:23.930914+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:03:23.930914+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'7600", 'ondisk_log_start': "39'7600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7705193, 'num_objects': 1882, 'num_object_clones': 0, 'num_object_copies': 3764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17646, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1653}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:26.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:03:26.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'17431", 'reported_seq': 17516, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:23.835633+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:03:23.835633+0000', 'last_peered': '2024-09-18T12:03:23.835633+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:03:23.835633+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'7400", 'ondisk_log_start': "39'7400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7238866, 'num_objects': 1769, 'num_object_clones': 0, 'num_object_copies': 3538, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17429, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1824}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:26.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:03:26.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'17694", 'reported_seq': 17783, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:23.941863+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:03:23.941863+0000', 'last_peered': '2024-09-18T12:03:23.941863+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:03:23.941863+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'7600", 'ondisk_log_start': "39'7600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7448996, 'num_objects': 1822, 'num_object_clones': 0, 'num_object_copies': 3644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17691, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1701}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:26.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:03:27.410 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:27.410 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30504/58652 objects degraded (52.008%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":607,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":29326,"data_bytes":120037471,"bytes_used":1355259904,"bytes_avail":288555032576,"bytes_total":289910292480,"degraded_objects":30504,"degraded_total":58652,"degraded_ratio":0.52008456659619451,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":230,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":527},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:03:27.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:03:27.441 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:28.182 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:28.198 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-09-18T12:03:27.244805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118927455,"num_objects":29055,"num_object_clones":0,"num_object_copies":58110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29055,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":266164,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12090,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160772,"ondisk_log_size":160772,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1304488,"kb_used_data":402640,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281811032,"statfs":{"total":289910292480,"available":288574496768,"internally_reserved":0,"allocated":412303360,"data_stored":411509056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,4,10,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19075072,"num_objects":-4657,"num_object_clones":0,"num_object_copies":-9314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4657,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002689"},"pg_stats":[{"pgid":"2.f","version":"43'17686","reported_seq":17761,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.964169+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:23.964169+0000","last_peered":"2024-09-18T12:03:23.964169+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:23.964169+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7577600,"num_objects":1850,"num_object_clones":0,"num_object_copies":3700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17686,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1749}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17617","reported_seq":17779,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:23.825039+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:23.825039+0000","last_peered":"2024-09-18T12:03:23.825039+0000","last_clean":"2024-09-18T12:03:23.825039+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:23.825039+0000","last_undegraded":"2024-09-18T12:03:23.825039+0000","last_fullsized":"2024-09-18T12:03:23.825039+0000","mapping_epoch":29,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7434240,"num_objects":1815,"num_object_clones":0,"num_object_copies":3630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14809,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17720","reported_seq":17910,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.791928+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:23.791928+0000","last_peered":"2024-09-18T12:03:23.791928+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:23.791928+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7524376,"num_objects":1838,"num_object_clones":0,"num_object_copies":3676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14578,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1698}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17458","reported_seq":17491,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:23.694057+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:23.694057+0000","last_peered":"2024-09-18T12:03:23.694057+0000","last_clean":"2024-09-18T12:03:23.694057+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:23.694057+0000","last_undegraded":"2024-09-18T12:03:23.694057+0000","last_fullsized":"2024-09-18T12:03:23.694057+0000","mapping_epoch":29,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7380992,"num_objects":1802,"num_object_clones":0,"num_object_copies":3604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17458,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17646","reported_seq":17841,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:23.924633+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:23.924633+0000","last_peered":"2024-09-18T12:03:23.924633+0000","last_clean":"2024-09-18T12:03:23.924633+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:23.924633+0000","last_undegraded":"2024-09-18T12:03:23.924633+0000","last_fullsized":"2024-09-18T12:03:23.924633+0000","mapping_epoch":29,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7376896,"num_objects":1801,"num_object_clones":0,"num_object_copies":3602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14811,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17583","reported_seq":17665,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.966801+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:23.966801+0000","last_peered":"2024-09-18T12:03:23.966801+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:23.966801+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7537874,"num_objects":1842,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17581,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1685}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17576","reported_seq":17667,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.911536+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:23.911536+0000","last_peered":"2024-09-18T12:03:23.911536+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:23.911536+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7357650,"num_objects":1798,"num_object_clones":0,"num_object_copies":3596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17574,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17806","reported_seq":18035,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:23.925625+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:23.925625+0000","last_peered":"2024-09-18T12:03:23.925625+0000","last_clean":"2024-09-18T12:03:23.925625+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:23.925625+0000","last_undegraded":"2024-09-18T12:03:23.925625+0000","last_fullsized":"2024-09-18T12:03:23.925625+0000","mapping_epoch":29,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7356416,"num_objects":1796,"num_object_clones":0,"num_object_copies":3592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14123,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17623","reported_seq":24755,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:25.716458+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:25.716458+0000","last_peered":"2024-09-18T12:03:25.716458+0000","last_clean":"2024-09-18T12:03:25.716458+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:25.716458+0000","last_undegraded":"2024-09-18T12:03:25.716458+0000","last_fullsized":"2024-09-18T12:03:25.716458+0000","mapping_epoch":38,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7300306,"num_objects":1784,"num_object_clones":0,"num_object_copies":3568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17621,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17627","reported_seq":20639,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.965465+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:23.965465+0000","last_peered":"2024-09-18T12:03:23.965465+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:23.965465+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7451858,"num_objects":1821,"num_object_clones":0,"num_object_copies":3642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17625,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-248}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17666","reported_seq":17726,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.790872+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:23.790872+0000","last_peered":"2024-09-18T12:03:23.790872+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:23.790872+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7361746,"num_objects":1799,"num_object_clones":0,"num_object_copies":3598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17664,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1751}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17612","reported_seq":25025,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:23.825493+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:23.825493+0000","last_peered":"2024-09-18T12:03:23.825493+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:23.825493+0000","last_undegraded":"2024-09-18T12:03:23.825493+0000","last_fullsized":"2024-09-18T12:03:23.825493+0000","mapping_epoch":42,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7725056,"num_objects":1886,"num_object_clones":0,"num_object_copies":3772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14092,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1886}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17647","reported_seq":17733,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.930914+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:23.930914+0000","last_peered":"2024-09-18T12:03:23.930914+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:23.930914+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7705193,"num_objects":1882,"num_object_clones":0,"num_object_copies":3764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17646,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17777","reported_seq":24927,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:25.611862+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:25.611862+0000","last_peered":"2024-09-18T12:03:25.611862+0000","last_clean":"2024-09-18T12:03:25.611862+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:25.611862+0000","last_undegraded":"2024-09-18T12:03:25.611862+0000","last_fullsized":"2024-09-18T12:03:25.611862+0000","mapping_epoch":40,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7149371,"num_objects":1748,"num_object_clones":0,"num_object_copies":3496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17774,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17431","reported_seq":17516,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.835633+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:23.835633+0000","last_peered":"2024-09-18T12:03:23.835633+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:23.835633+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'7400","ondisk_log_start":"39'7400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7238866,"num_objects":1769,"num_object_clones":0,"num_object_copies":3538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17429,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17694","reported_seq":17783,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:23.941863+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:23.941863+0000","last_peered":"2024-09-18T12:03:23.941863+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:23.941863+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7448996,"num_objects":1822,"num_object_clones":0,"num_object_copies":3644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17691,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":118927436,"num_objects":29053,"num_object_clones":0,"num_object_copies":58106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":266162,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12089,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":410529792,"data_stored":410465984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160769,"ondisk_log_size":160769,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051698,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474652,"kb_used_data":172628,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93897188,"statfs":{"total":96636764160,"available":96150720512,"internally_reserved":0,"allocated":176771072,"data_stored":176522437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.68200000000000005},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":1.3340000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855461,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383776,"kb_used_data":85912,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93988064,"statfs":{"total":96636764160,"available":96243777536,"internally_reserved":0,"allocated":87973888,"data_stored":87676404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.218},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.47699999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446060,"kb_used_data":144100,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93925780,"statfs":{"total":96636764160,"available":96179998720,"internally_reserved":0,"allocated":147558400,"data_stored":147310215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20999999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147128320,"data_stored":147124841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87425024,"data_stored":87372246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":175976448,"data_stored":175968897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:31.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:31.843 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:31.855 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:31.879 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-18T12:03:31.245637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":110010463,"num_objects":26878,"num_object_clones":0,"num_object_copies":53756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268341,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12170,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1303760,"kb_used_data":385272,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281811760,"statfs":{"total":289910292480,"available":288575242240,"internally_reserved":0,"allocated":394518528,"data_stored":393724224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,0,7,8,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19046400,"num_objects":-4650,"num_object_clones":0,"num_object_copies":-9300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4650,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002614"},"pg_stats":[{"pgid":"2.f","version":"43'17821","reported_seq":17896,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.855358+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:28.855358+0000","last_peered":"2024-09-18T12:03:28.855358+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:28.855358+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7024640,"num_objects":1715,"num_object_clones":0,"num_object_copies":3430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17821,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1884}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17760","reported_seq":17922,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:28.780623+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:28.780623+0000","last_peered":"2024-09-18T12:03:28.780623+0000","last_clean":"2024-09-18T12:03:28.780623+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:28.780623+0000","last_undegraded":"2024-09-18T12:03:28.780623+0000","last_fullsized":"2024-09-18T12:03:28.780623+0000","mapping_epoch":29,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6848512,"num_objects":1672,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14952,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17859","reported_seq":18048,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.822704+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:28.822704+0000","last_peered":"2024-09-18T12:03:28.822704+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:28.822704+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6955032,"num_objects":1699,"num_object_clones":0,"num_object_copies":3398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14717,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1837}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17593","reported_seq":17626,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:28.843838+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:28.843838+0000","last_peered":"2024-09-18T12:03:28.843838+0000","last_clean":"2024-09-18T12:03:28.843838+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:28.843838+0000","last_undegraded":"2024-09-18T12:03:28.843838+0000","last_fullsized":"2024-09-18T12:03:28.843838+0000","mapping_epoch":29,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6828032,"num_objects":1667,"num_object_clones":0,"num_object_copies":3334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17593,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17787","reported_seq":17982,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:28.994728+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:28.994728+0000","last_peered":"2024-09-18T12:03:28.994728+0000","last_clean":"2024-09-18T12:03:28.994728+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:28.994728+0000","last_undegraded":"2024-09-18T12:03:28.994728+0000","last_fullsized":"2024-09-18T12:03:28.994728+0000","mapping_epoch":29,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6799360,"num_objects":1660,"num_object_clones":0,"num_object_copies":3320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14952,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17729","reported_seq":17814,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.960181+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:28.960181+0000","last_peered":"2024-09-18T12:03:28.960181+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:28.960181+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6939858,"num_objects":1696,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17727,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17702","reported_seq":17793,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.953091+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:28.953091+0000","last_peered":"2024-09-18T12:03:28.953091+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:28.953091+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6841554,"num_objects":1672,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17700,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17929","reported_seq":18158,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:28.970870+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:28.970870+0000","last_peered":"2024-09-18T12:03:28.970870+0000","last_clean":"2024-09-18T12:03:28.970870+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:28.970870+0000","last_undegraded":"2024-09-18T12:03:28.970870+0000","last_fullsized":"2024-09-18T12:03:28.970870+0000","mapping_epoch":29,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6852608,"num_objects":1673,"num_object_clones":0,"num_object_copies":3346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14246,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17772","reported_seq":24904,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:30.711493+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:30.711493+0000","last_peered":"2024-09-18T12:03:30.711493+0000","last_clean":"2024-09-18T12:03:30.711493+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:30.711493+0000","last_undegraded":"2024-09-18T12:03:30.711493+0000","last_fullsized":"2024-09-18T12:03:30.711493+0000","mapping_epoch":38,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6690002,"num_objects":1635,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17770,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17757","reported_seq":20930,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.996477+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:28.996477+0000","last_peered":"2024-09-18T12:03:28.996477+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:28.996477+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6919378,"num_objects":1691,"num_object_clones":0,"num_object_copies":3382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17755,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1547,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-298}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17797","reported_seq":17857,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.686643+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:28.686643+0000","last_peered":"2024-09-18T12:03:28.686643+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:28.686643+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6825170,"num_objects":1668,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17795,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1882}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17761","reported_seq":25174,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:28.808535+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:28.808535+0000","last_peered":"2024-09-18T12:03:28.808535+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:28.808535+0000","last_undegraded":"2024-09-18T12:03:28.808535+0000","last_fullsized":"2024-09-18T12:03:28.808535+0000","mapping_epoch":42,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7114752,"num_objects":1737,"num_object_clones":0,"num_object_copies":3474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14241,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17785","reported_seq":17871,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.970877+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:28.970877+0000","last_peered":"2024-09-18T12:03:28.970877+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:28.970877+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7139945,"num_objects":1744,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17784,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17901","reported_seq":25051,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:30.787690+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:30.787690+0000","last_peered":"2024-09-18T12:03:30.787690+0000","last_clean":"2024-09-18T12:03:30.787690+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:30.787690+0000","last_undegraded":"2024-09-18T12:03:30.787690+0000","last_fullsized":"2024-09-18T12:03:30.787690+0000","mapping_epoch":40,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6641467,"num_objects":1624,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17898,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17550","reported_seq":17630,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.881300+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:28.881300+0000","last_peered":"2024-09-18T12:03:28.881300+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:28.881300+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6751442,"num_objects":1650,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17548,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17843","reported_seq":17932,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.923673+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:28.923673+0000","last_peered":"2024-09-18T12:03:28.923673+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:28.923673+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6838692,"num_objects":1673,"num_object_clones":0,"num_object_copies":3346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17840,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1850}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":110010444,"num_objects":26876,"num_object_clones":0,"num_object_copies":53752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26876,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268339,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12169,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":392699904,"data_stored":392636096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160846,"ondisk_log_size":160846,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051699,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474488,"kb_used_data":168304,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93897352,"statfs":{"total":96636764160,"available":96150888448,"internally_reserved":0,"allocated":172343296,"data_stored":172094661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.22800000000000001},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":2.2450000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":2.2050000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.436}]}]},{"osd":1,"up_from":35,"seq":150323855462,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385572,"kb_used_data":79388,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93986268,"statfs":{"total":96636764160,"available":96241938432,"internally_reserved":0,"allocated":81293312,"data_stored":80995828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.183},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.442}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.40400000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443700,"kb_used_data":137580,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93928140,"statfs":{"total":96636764160,"available":96182415360,"internally_reserved":0,"allocated":140881920,"data_stored":140633735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,1,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":140492800,"data_stored":140489321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":80687104,"data_stored":80634326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171520000,"data_stored":171512449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:31.881 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:32.515 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:32.535 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-18T12:03:31.245637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":110010463,"num_objects":26878,"num_object_clones":0,"num_object_copies":53756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268341,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12170,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1303760,"kb_used_data":385272,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281811760,"statfs":{"total":289910292480,"available":288575242240,"internally_reserved":0,"allocated":394518528,"data_stored":393724224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,0,7,8,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19046400,"num_objects":-4650,"num_object_clones":0,"num_object_copies":-9300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4650,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002614"},"pg_stats":[{"pgid":"2.f","version":"43'17821","reported_seq":17896,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.855358+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:28.855358+0000","last_peered":"2024-09-18T12:03:28.855358+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:28.855358+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7024640,"num_objects":1715,"num_object_clones":0,"num_object_copies":3430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17821,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1884}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17760","reported_seq":17922,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:28.780623+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:28.780623+0000","last_peered":"2024-09-18T12:03:28.780623+0000","last_clean":"2024-09-18T12:03:28.780623+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:28.780623+0000","last_undegraded":"2024-09-18T12:03:28.780623+0000","last_fullsized":"2024-09-18T12:03:28.780623+0000","mapping_epoch":29,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6848512,"num_objects":1672,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14952,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17859","reported_seq":18048,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.822704+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:28.822704+0000","last_peered":"2024-09-18T12:03:28.822704+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:28.822704+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6955032,"num_objects":1699,"num_object_clones":0,"num_object_copies":3398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14717,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1837}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17593","reported_seq":17626,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:28.843838+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:28.843838+0000","last_peered":"2024-09-18T12:03:28.843838+0000","last_clean":"2024-09-18T12:03:28.843838+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:28.843838+0000","last_undegraded":"2024-09-18T12:03:28.843838+0000","last_fullsized":"2024-09-18T12:03:28.843838+0000","mapping_epoch":29,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6828032,"num_objects":1667,"num_object_clones":0,"num_object_copies":3334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17593,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17787","reported_seq":17982,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:28.994728+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:28.994728+0000","last_peered":"2024-09-18T12:03:28.994728+0000","last_clean":"2024-09-18T12:03:28.994728+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:28.994728+0000","last_undegraded":"2024-09-18T12:03:28.994728+0000","last_fullsized":"2024-09-18T12:03:28.994728+0000","mapping_epoch":29,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6799360,"num_objects":1660,"num_object_clones":0,"num_object_copies":3320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14952,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17729","reported_seq":17814,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.960181+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:28.960181+0000","last_peered":"2024-09-18T12:03:28.960181+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:28.960181+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6939858,"num_objects":1696,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17727,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17702","reported_seq":17793,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.953091+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:28.953091+0000","last_peered":"2024-09-18T12:03:28.953091+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:28.953091+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6841554,"num_objects":1672,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17700,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17929","reported_seq":18158,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:28.970870+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:28.970870+0000","last_peered":"2024-09-18T12:03:28.970870+0000","last_clean":"2024-09-18T12:03:28.970870+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:28.970870+0000","last_undegraded":"2024-09-18T12:03:28.970870+0000","last_fullsized":"2024-09-18T12:03:28.970870+0000","mapping_epoch":29,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6852608,"num_objects":1673,"num_object_clones":0,"num_object_copies":3346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14246,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17772","reported_seq":24904,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:30.711493+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:30.711493+0000","last_peered":"2024-09-18T12:03:30.711493+0000","last_clean":"2024-09-18T12:03:30.711493+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:30.711493+0000","last_undegraded":"2024-09-18T12:03:30.711493+0000","last_fullsized":"2024-09-18T12:03:30.711493+0000","mapping_epoch":38,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6690002,"num_objects":1635,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17770,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17757","reported_seq":20930,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.996477+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:28.996477+0000","last_peered":"2024-09-18T12:03:28.996477+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:28.996477+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6919378,"num_objects":1691,"num_object_clones":0,"num_object_copies":3382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17755,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1547,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-298}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17797","reported_seq":17857,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.686643+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:28.686643+0000","last_peered":"2024-09-18T12:03:28.686643+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:28.686643+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6825170,"num_objects":1668,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17795,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1882}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17761","reported_seq":25174,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:28.808535+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:28.808535+0000","last_peered":"2024-09-18T12:03:28.808535+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:28.808535+0000","last_undegraded":"2024-09-18T12:03:28.808535+0000","last_fullsized":"2024-09-18T12:03:28.808535+0000","mapping_epoch":42,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7114752,"num_objects":1737,"num_object_clones":0,"num_object_copies":3474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14241,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17785","reported_seq":17871,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.970877+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:28.970877+0000","last_peered":"2024-09-18T12:03:28.970877+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:28.970877+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7139945,"num_objects":1744,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17784,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17901","reported_seq":25051,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:30.787690+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:30.787690+0000","last_peered":"2024-09-18T12:03:30.787690+0000","last_clean":"2024-09-18T12:03:30.787690+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:30.787690+0000","last_undegraded":"2024-09-18T12:03:30.787690+0000","last_fullsized":"2024-09-18T12:03:30.787690+0000","mapping_epoch":40,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6641467,"num_objects":1624,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17898,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17550","reported_seq":17630,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.881300+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:28.881300+0000","last_peered":"2024-09-18T12:03:28.881300+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:28.881300+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6751442,"num_objects":1650,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17548,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17843","reported_seq":17932,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.923673+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:28.923673+0000","last_peered":"2024-09-18T12:03:28.923673+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:28.923673+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6838692,"num_objects":1673,"num_object_clones":0,"num_object_copies":3346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17840,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1850}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":110010444,"num_objects":26876,"num_object_clones":0,"num_object_copies":53752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26876,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268339,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12169,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":392699904,"data_stored":392636096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160846,"ondisk_log_size":160846,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051699,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474488,"kb_used_data":168304,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93897352,"statfs":{"total":96636764160,"available":96150888448,"internally_reserved":0,"allocated":172343296,"data_stored":172094661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.22800000000000001},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":2.2450000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":2.2050000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.436}]}]},{"osd":1,"up_from":35,"seq":150323855462,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385572,"kb_used_data":79388,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93986268,"statfs":{"total":96636764160,"available":96241938432,"internally_reserved":0,"allocated":81293312,"data_stored":80995828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.183},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.442}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.40400000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443700,"kb_used_data":137580,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93928140,"statfs":{"total":96636764160,"available":96182415360,"internally_reserved":0,"allocated":140881920,"data_stored":140633735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,1,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":140492800,"data_stored":140489321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":80687104,"data_stored":80634326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171520000,"data_stored":171512449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:32.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:03:32.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'17821", 'reported_seq': 17896, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:28.855358+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:03:28.855358+0000', 'last_peered': '2024-09-18T12:03:28.855358+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:03:28.855358+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'7800", 'ondisk_log_start': "39'7800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7024640, 'num_objects': 1715, 'num_object_clones': 0, 'num_object_copies': 3430, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17821, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1884}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:32.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:03:32.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'17859", 'reported_seq': 18048, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:28.822704+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:03:28.822704+0000', 'last_peered': '2024-09-18T12:03:28.822704+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:03:28.822704+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'7800", 'ondisk_log_start': "39'7800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6955032, 'num_objects': 1699, 'num_object_clones': 0, 'num_object_copies': 3398, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1699, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14717, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:32.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:03:32.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'17729", 'reported_seq': 17814, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:28.960181+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:03:28.960181+0000', 'last_peered': '2024-09-18T12:03:28.960181+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:03:28.960181+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'7700", 'ondisk_log_start': "39'7700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6939858, 'num_objects': 1696, 'num_object_clones': 0, 'num_object_copies': 3392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17727, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1831}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:32.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:03:32.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'17702", 'reported_seq': 17793, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:28.953091+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:03:28.953091+0000', 'last_peered': '2024-09-18T12:03:28.953091+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:03:28.953091+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'7700", 'ondisk_log_start': "39'7700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6841554, 'num_objects': 1672, 'num_object_clones': 0, 'num_object_copies': 3344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17700, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1887}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:32.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:03:32.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'17757", 'reported_seq': 20930, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:28.996477+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:03:28.996477+0000', 'last_peered': '2024-09-18T12:03:28.996477+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:03:28.996477+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'7700", 'ondisk_log_start': "39'7700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6919378, 'num_objects': 1691, 'num_object_clones': 0, 'num_object_copies': 3382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17755, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1547, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -298}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:32.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:03:32.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'17797", 'reported_seq': 17857, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:28.686643+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:03:28.686643+0000', 'last_peered': '2024-09-18T12:03:28.686643+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:03:28.686643+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'7700", 'ondisk_log_start': "39'7700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6825170, 'num_objects': 1668, 'num_object_clones': 0, 'num_object_copies': 3336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17795, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1882}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:32.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:03:32.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'17761", 'reported_seq': 25174, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:03:28.808535+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:03:28.808535+0000', 'last_peered': '2024-09-18T12:03:28.808535+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:03:28.808535+0000', 'last_undegraded': '2024-09-18T12:03:28.808535+0000', 'last_fullsized': '2024-09-18T12:03:28.808535+0000', 'mapping_epoch': 42, 'log_start': "39'7700", 'ondisk_log_start': "39'7700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7114752, 'num_objects': 1737, 'num_object_clones': 0, 'num_object_copies': 3474, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1737, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14241, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1737}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:32.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:03:32.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'17785", 'reported_seq': 17871, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:28.970877+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:03:28.970877+0000', 'last_peered': '2024-09-18T12:03:28.970877+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:03:28.970877+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'7700", 'ondisk_log_start': "39'7700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7139945, 'num_objects': 1744, 'num_object_clones': 0, 'num_object_copies': 3488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17784, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1791}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:32.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:03:32.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'17550", 'reported_seq': 17630, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:28.881300+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:03:28.881300+0000', 'last_peered': '2024-09-18T12:03:28.881300+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:03:28.881300+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'7500", 'ondisk_log_start': "39'7500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6751442, 'num_objects': 1650, 'num_object_clones': 0, 'num_object_copies': 3300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1650, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17548, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1943}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:32.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:03:32.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'17843", 'reported_seq': 17932, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:28.923673+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:03:28.923673+0000', 'last_peered': '2024-09-18T12:03:28.923673+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:03:28.923673+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'7800", 'ondisk_log_start': "39'7800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6838692, 'num_objects': 1673, 'num_object_clones': 0, 'num_object_copies': 3346, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1673, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17840, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1850}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:32.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:03:33.546 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:33.546 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30424/53756 objects degraded (56.596%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":613,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":26878,"data_bytes":110010463,"bytes_used":1335050240,"bytes_avail":288575242240,"bytes_total":289910292480,"degraded_objects":30424,"degraded_total":53756,"degraded_ratio":0.56596472951856536,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":387},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:03:33.590 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:03:33.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:34.358 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:34.364 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-18T12:03:33.246094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":110010463,"num_objects":26878,"num_object_clones":0,"num_object_copies":53756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26878,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268341,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12170,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1303760,"kb_used_data":385272,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281811760,"statfs":{"total":289910292480,"available":288575242240,"internally_reserved":0,"allocated":394518528,"data_stored":393724224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,0,7,8,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17891328,"num_objects":-4368,"num_object_clones":0,"num_object_copies":-8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4368,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002618"},"pg_stats":[{"pgid":"2.f","version":"43'17821","reported_seq":17896,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.855358+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:28.855358+0000","last_peered":"2024-09-18T12:03:28.855358+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:28.855358+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7024640,"num_objects":1715,"num_object_clones":0,"num_object_copies":3430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17821,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1884}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17760","reported_seq":17922,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:28.780623+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:28.780623+0000","last_peered":"2024-09-18T12:03:28.780623+0000","last_clean":"2024-09-18T12:03:28.780623+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:28.780623+0000","last_undegraded":"2024-09-18T12:03:28.780623+0000","last_fullsized":"2024-09-18T12:03:28.780623+0000","mapping_epoch":29,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6848512,"num_objects":1672,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14952,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'17859","reported_seq":18048,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.822704+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:28.822704+0000","last_peered":"2024-09-18T12:03:28.822704+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:28.822704+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6955032,"num_objects":1699,"num_object_clones":0,"num_object_copies":3398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14717,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1837}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17593","reported_seq":17626,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:28.843838+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:28.843838+0000","last_peered":"2024-09-18T12:03:28.843838+0000","last_clean":"2024-09-18T12:03:28.843838+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:28.843838+0000","last_undegraded":"2024-09-18T12:03:28.843838+0000","last_fullsized":"2024-09-18T12:03:28.843838+0000","mapping_epoch":29,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6828032,"num_objects":1667,"num_object_clones":0,"num_object_copies":3334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17593,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17787","reported_seq":17982,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:28.994728+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:28.994728+0000","last_peered":"2024-09-18T12:03:28.994728+0000","last_clean":"2024-09-18T12:03:28.994728+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:28.994728+0000","last_undegraded":"2024-09-18T12:03:28.994728+0000","last_fullsized":"2024-09-18T12:03:28.994728+0000","mapping_epoch":29,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6799360,"num_objects":1660,"num_object_clones":0,"num_object_copies":3320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14952,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17729","reported_seq":17814,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.960181+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:28.960181+0000","last_peered":"2024-09-18T12:03:28.960181+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:28.960181+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6939858,"num_objects":1696,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17727,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1831}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17702","reported_seq":17793,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.953091+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:28.953091+0000","last_peered":"2024-09-18T12:03:28.953091+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:28.953091+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6841554,"num_objects":1672,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17700,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'17929","reported_seq":18158,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:28.970870+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:28.970870+0000","last_peered":"2024-09-18T12:03:28.970870+0000","last_clean":"2024-09-18T12:03:28.970870+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:28.970870+0000","last_undegraded":"2024-09-18T12:03:28.970870+0000","last_fullsized":"2024-09-18T12:03:28.970870+0000","mapping_epoch":29,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6852608,"num_objects":1673,"num_object_clones":0,"num_object_copies":3346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14246,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17772","reported_seq":24904,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:30.711493+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:30.711493+0000","last_peered":"2024-09-18T12:03:30.711493+0000","last_clean":"2024-09-18T12:03:30.711493+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:30.711493+0000","last_undegraded":"2024-09-18T12:03:30.711493+0000","last_fullsized":"2024-09-18T12:03:30.711493+0000","mapping_epoch":38,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6690002,"num_objects":1635,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17770,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17757","reported_seq":20930,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.996477+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:28.996477+0000","last_peered":"2024-09-18T12:03:28.996477+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:28.996477+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6919378,"num_objects":1691,"num_object_clones":0,"num_object_copies":3382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17755,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1547,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-298}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17797","reported_seq":17857,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.686643+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:28.686643+0000","last_peered":"2024-09-18T12:03:28.686643+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:28.686643+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6825170,"num_objects":1668,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17795,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1882}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17761","reported_seq":25174,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:28.808535+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:28.808535+0000","last_peered":"2024-09-18T12:03:28.808535+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:28.808535+0000","last_undegraded":"2024-09-18T12:03:28.808535+0000","last_fullsized":"2024-09-18T12:03:28.808535+0000","mapping_epoch":42,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7114752,"num_objects":1737,"num_object_clones":0,"num_object_copies":3474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14241,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1737}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17785","reported_seq":17871,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.970877+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:28.970877+0000","last_peered":"2024-09-18T12:03:28.970877+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:28.970877+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7139945,"num_objects":1744,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17784,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'17901","reported_seq":25051,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:30.787690+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:30.787690+0000","last_peered":"2024-09-18T12:03:30.787690+0000","last_clean":"2024-09-18T12:03:30.787690+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:30.787690+0000","last_undegraded":"2024-09-18T12:03:30.787690+0000","last_fullsized":"2024-09-18T12:03:30.787690+0000","mapping_epoch":40,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6641467,"num_objects":1624,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17898,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17550","reported_seq":17630,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.881300+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:28.881300+0000","last_peered":"2024-09-18T12:03:28.881300+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:28.881300+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'7500","ondisk_log_start":"39'7500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6751442,"num_objects":1650,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17548,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17843","reported_seq":17932,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:28.923673+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:28.923673+0000","last_peered":"2024-09-18T12:03:28.923673+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:28.923673+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6838692,"num_objects":1673,"num_object_clones":0,"num_object_copies":3346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17840,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1850}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":110010444,"num_objects":26876,"num_object_clones":0,"num_object_copies":53752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26876,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268339,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12169,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":392699904,"data_stored":392636096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160846,"ondisk_log_size":160846,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051699,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474488,"kb_used_data":168304,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93897352,"statfs":{"total":96636764160,"available":96150888448,"internally_reserved":0,"allocated":172343296,"data_stored":172094661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:02:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.81399999999999995,"15min":0.56699999999999995},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.3390000000000004,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.22800000000000001},{"interface":"front","average":{"1min":2.7080000000000002,"5min":1.706,"15min":0.91800000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":9.5169999999999995,"5min":10.795999999999999,"15min":10.795999999999999},"last":2.2450000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":2.2050000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.436}]}]},{"osd":1,"up_from":35,"seq":150323855462,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385572,"kb_used_data":79388,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93986268,"statfs":{"total":96636764160,"available":96241938432,"internally_reserved":0,"allocated":81293312,"data_stored":80995828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.183},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.442}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.40400000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443700,"kb_used_data":137580,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93928140,"statfs":{"total":96636764160,"available":96182415360,"internally_reserved":0,"allocated":140881920,"data_stored":140633735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,1,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":140492800,"data_stored":140489321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":80687104,"data_stored":80634326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171520000,"data_stored":171512449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:37.366 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:38.162 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:38.178 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-18T12:03:37.246965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101077087,"num_objects":24697,"num_object_clones":0,"num_object_copies":49394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24697,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270522,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12252,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160730,"ondisk_log_size":160730,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1299892,"kb_used_data":368924,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":281815628,"statfs":{"total":289910292480,"available":288579203072,"internally_reserved":0,"allocated":377778176,"data_stored":376983872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,5,11,5,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18960384,"num_objects":-4629,"num_object_clones":0,"num_object_copies":-9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4629,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002652"},"pg_stats":[{"pgid":"2.f","version":"43'17961","reported_seq":18036,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.954129+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:33.954129+0000","last_peered":"2024-09-18T12:03:33.954129+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:33.954129+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6451200,"num_objects":1575,"num_object_clones":0,"num_object_copies":3150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17961,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17894","reported_seq":18056,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:33.851224+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:33.851224+0000","last_peered":"2024-09-18T12:03:33.851224+0000","last_clean":"2024-09-18T12:03:33.851224+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:33.851224+0000","last_undegraded":"2024-09-18T12:03:33.851224+0000","last_fullsized":"2024-09-18T12:03:33.851224+0000","mapping_epoch":29,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6299648,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15086,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18004","reported_seq":18194,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.832298+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:33.832298+0000","last_peered":"2024-09-18T12:03:33.832298+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:33.832298+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6361112,"num_objects":1554,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14862,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1982}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17741","reported_seq":17774,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:33.801661+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:33.801661+0000","last_peered":"2024-09-18T12:03:33.801661+0000","last_clean":"2024-09-18T12:03:33.801661+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:33.801661+0000","last_undegraded":"2024-09-18T12:03:33.801661+0000","last_fullsized":"2024-09-18T12:03:33.801661+0000","mapping_epoch":29,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6221824,"num_objects":1519,"num_object_clones":0,"num_object_copies":3038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17741,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17925","reported_seq":18120,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:33.966638+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:33.966638+0000","last_peered":"2024-09-18T12:03:33.966638+0000","last_clean":"2024-09-18T12:03:33.966638+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:33.966638+0000","last_undegraded":"2024-09-18T12:03:33.966638+0000","last_fullsized":"2024-09-18T12:03:33.966638+0000","mapping_epoch":29,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6234112,"num_objects":1522,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15090,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17865","reported_seq":17950,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.899864+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:33.899864+0000","last_peered":"2024-09-18T12:03:33.899864+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:33.899864+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6382802,"num_objects":1560,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17863,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1967}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17834","reported_seq":17925,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:34.010480+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:34.010480+0000","last_peered":"2024-09-18T12:03:34.010480+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:34.010480+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6300882,"num_objects":1540,"num_object_clones":0,"num_object_copies":3080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17832,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18071","reported_seq":18300,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:33.868081+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:33.868081+0000","last_peered":"2024-09-18T12:03:33.868081+0000","last_clean":"2024-09-18T12:03:33.868081+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:33.868081+0000","last_undegraded":"2024-09-18T12:03:33.868081+0000","last_fullsized":"2024-09-18T12:03:33.868081+0000","mapping_epoch":29,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6270976,"num_objects":1531,"num_object_clones":0,"num_object_copies":3062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14388,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17902","reported_seq":25034,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:35.728371+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:35.728371+0000","last_peered":"2024-09-18T12:03:35.728371+0000","last_clean":"2024-09-18T12:03:35.728371+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:35.728371+0000","last_undegraded":"2024-09-18T12:03:35.728371+0000","last_fullsized":"2024-09-18T12:03:35.728371+0000","mapping_epoch":38,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6157522,"num_objects":1505,"num_object_clones":0,"num_object_copies":3010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17900,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17879","reported_seq":21215,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:34.002279+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:34.002279+0000","last_peered":"2024-09-18T12:03:34.002279+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:34.002279+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6419666,"num_objects":1569,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17877,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1629,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-338}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17913","reported_seq":17973,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.844553+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:33.844553+0000","last_peered":"2024-09-18T12:03:33.844553+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:33.844553+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6350034,"num_objects":1552,"num_object_clones":0,"num_object_copies":3104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17911,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1998}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17903","reported_seq":25316,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:33.798936+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:33.798936+0000","last_peered":"2024-09-18T12:03:33.798936+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:33.798936+0000","last_undegraded":"2024-09-18T12:03:33.798936+0000","last_fullsized":"2024-09-18T12:03:33.798936+0000","mapping_epoch":42,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6533120,"num_objects":1595,"num_object_clones":0,"num_object_copies":3190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14383,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1595}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17921","reported_seq":18007,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.937175+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:33.937175+0000","last_peered":"2024-09-18T12:03:33.937175+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:33.937175+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6582889,"num_objects":1608,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17920,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1927}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18042","reported_seq":25192,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:35.823213+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:35.823213+0000","last_peered":"2024-09-18T12:03:35.823213+0000","last_clean":"2024-09-18T12:03:35.823213+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:35.823213+0000","last_undegraded":"2024-09-18T12:03:35.823213+0000","last_fullsized":"2024-09-18T12:03:35.823213+0000","mapping_epoch":40,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6063931,"num_objects":1483,"num_object_clones":0,"num_object_copies":2966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18039,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17685","reported_seq":17770,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.952370+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:33.952370+0000","last_peered":"2024-09-18T12:03:33.952370+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:33.952370+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6198482,"num_objects":1515,"num_object_clones":0,"num_object_copies":3030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17683,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17987","reported_seq":18076,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.971573+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:33.971573+0000","last_peered":"2024-09-18T12:03:33.971573+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:33.971573+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6248868,"num_objects":1529,"num_object_clones":0,"num_object_copies":3058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17984,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1994}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":101077068,"num_objects":24695,"num_object_clones":0,"num_object_copies":49390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24695,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270520,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12251,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":374603776,"data_stored":374539968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160727,"ondisk_log_size":160727,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051700,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474172,"kb_used_data":163828,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93897668,"statfs":{"total":96636764160,"available":96151212032,"internally_reserved":0,"allocated":167759872,"data_stored":167511237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.27800000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":2.5169999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":1.347},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":8.0449999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855463,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383212,"kb_used_data":72868,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93988628,"statfs":{"total":96636764160,"available":96244355072,"internally_reserved":0,"allocated":74616832,"data_stored":74319348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,3,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.26300000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.23300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":442508,"kb_used_data":132228,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93929332,"statfs":{"total":96636764160,"available":96183635968,"internally_reserved":0,"allocated":135401472,"data_stored":135153287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.219},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.29499999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":133750784,"data_stored":133747305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73904128,"data_stored":73851350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":166948864,"data_stored":166941313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:38.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:38.923 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:38.962 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:38.972 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-18T12:03:37.246965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101077087,"num_objects":24697,"num_object_clones":0,"num_object_copies":49394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24697,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270522,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12252,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160730,"ondisk_log_size":160730,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1299892,"kb_used_data":368924,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":281815628,"statfs":{"total":289910292480,"available":288579203072,"internally_reserved":0,"allocated":377778176,"data_stored":376983872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,5,11,5,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18960384,"num_objects":-4629,"num_object_clones":0,"num_object_copies":-9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4629,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002652"},"pg_stats":[{"pgid":"2.f","version":"43'17961","reported_seq":18036,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.954129+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:33.954129+0000","last_peered":"2024-09-18T12:03:33.954129+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:33.954129+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6451200,"num_objects":1575,"num_object_clones":0,"num_object_copies":3150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17961,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'17894","reported_seq":18056,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:33.851224+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:33.851224+0000","last_peered":"2024-09-18T12:03:33.851224+0000","last_clean":"2024-09-18T12:03:33.851224+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:33.851224+0000","last_undegraded":"2024-09-18T12:03:33.851224+0000","last_fullsized":"2024-09-18T12:03:33.851224+0000","mapping_epoch":29,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6299648,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15086,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18004","reported_seq":18194,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.832298+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:33.832298+0000","last_peered":"2024-09-18T12:03:33.832298+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:33.832298+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6361112,"num_objects":1554,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14862,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1982}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17741","reported_seq":17774,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:33.801661+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:33.801661+0000","last_peered":"2024-09-18T12:03:33.801661+0000","last_clean":"2024-09-18T12:03:33.801661+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:33.801661+0000","last_undegraded":"2024-09-18T12:03:33.801661+0000","last_fullsized":"2024-09-18T12:03:33.801661+0000","mapping_epoch":29,"log_start":"39'7700","ondisk_log_start":"39'7700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6221824,"num_objects":1519,"num_object_clones":0,"num_object_copies":3038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17741,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'17925","reported_seq":18120,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:33.966638+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:33.966638+0000","last_peered":"2024-09-18T12:03:33.966638+0000","last_clean":"2024-09-18T12:03:33.966638+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:33.966638+0000","last_undegraded":"2024-09-18T12:03:33.966638+0000","last_fullsized":"2024-09-18T12:03:33.966638+0000","mapping_epoch":29,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6234112,"num_objects":1522,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15090,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'17865","reported_seq":17950,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.899864+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:33.899864+0000","last_peered":"2024-09-18T12:03:33.899864+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:33.899864+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6382802,"num_objects":1560,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17863,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1967}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17834","reported_seq":17925,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:34.010480+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:34.010480+0000","last_peered":"2024-09-18T12:03:34.010480+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:34.010480+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6300882,"num_objects":1540,"num_object_clones":0,"num_object_copies":3080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17832,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18071","reported_seq":18300,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:33.868081+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:33.868081+0000","last_peered":"2024-09-18T12:03:33.868081+0000","last_clean":"2024-09-18T12:03:33.868081+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:33.868081+0000","last_undegraded":"2024-09-18T12:03:33.868081+0000","last_fullsized":"2024-09-18T12:03:33.868081+0000","mapping_epoch":29,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6270976,"num_objects":1531,"num_object_clones":0,"num_object_copies":3062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14388,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17902","reported_seq":25034,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:35.728371+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:35.728371+0000","last_peered":"2024-09-18T12:03:35.728371+0000","last_clean":"2024-09-18T12:03:35.728371+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:35.728371+0000","last_undegraded":"2024-09-18T12:03:35.728371+0000","last_fullsized":"2024-09-18T12:03:35.728371+0000","mapping_epoch":38,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6157522,"num_objects":1505,"num_object_clones":0,"num_object_copies":3010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17900,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'17879","reported_seq":21215,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:34.002279+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:34.002279+0000","last_peered":"2024-09-18T12:03:34.002279+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:34.002279+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6419666,"num_objects":1569,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17877,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1629,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-338}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'17913","reported_seq":17973,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.844553+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:33.844553+0000","last_peered":"2024-09-18T12:03:33.844553+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:33.844553+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6350034,"num_objects":1552,"num_object_clones":0,"num_object_copies":3104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17911,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1998}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'17903","reported_seq":25316,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:33.798936+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:33.798936+0000","last_peered":"2024-09-18T12:03:33.798936+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:33.798936+0000","last_undegraded":"2024-09-18T12:03:33.798936+0000","last_fullsized":"2024-09-18T12:03:33.798936+0000","mapping_epoch":42,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6533120,"num_objects":1595,"num_object_clones":0,"num_object_copies":3190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14383,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1595}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'17921","reported_seq":18007,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.937175+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:33.937175+0000","last_peered":"2024-09-18T12:03:33.937175+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:33.937175+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6582889,"num_objects":1608,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17920,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1927}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18042","reported_seq":25192,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:35.823213+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:35.823213+0000","last_peered":"2024-09-18T12:03:35.823213+0000","last_clean":"2024-09-18T12:03:35.823213+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:35.823213+0000","last_undegraded":"2024-09-18T12:03:35.823213+0000","last_fullsized":"2024-09-18T12:03:35.823213+0000","mapping_epoch":40,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6063931,"num_objects":1483,"num_object_clones":0,"num_object_copies":2966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18039,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17685","reported_seq":17770,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.952370+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:33.952370+0000","last_peered":"2024-09-18T12:03:33.952370+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:33.952370+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'7600","ondisk_log_start":"39'7600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6198482,"num_objects":1515,"num_object_clones":0,"num_object_copies":3030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17683,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2078}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'17987","reported_seq":18076,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:33.971573+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:33.971573+0000","last_peered":"2024-09-18T12:03:33.971573+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:33.971573+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6248868,"num_objects":1529,"num_object_clones":0,"num_object_copies":3058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17984,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1994}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":101077068,"num_objects":24695,"num_object_clones":0,"num_object_copies":49390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24695,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270520,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12251,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":374603776,"data_stored":374539968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160727,"ondisk_log_size":160727,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051700,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474172,"kb_used_data":163828,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93897668,"statfs":{"total":96636764160,"available":96151212032,"internally_reserved":0,"allocated":167759872,"data_stored":167511237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.27800000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":2.5169999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":1.347},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":8.0449999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855463,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383212,"kb_used_data":72868,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93988628,"statfs":{"total":96636764160,"available":96244355072,"internally_reserved":0,"allocated":74616832,"data_stored":74319348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,3,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.26300000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.23300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":442508,"kb_used_data":132228,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93929332,"statfs":{"total":96636764160,"available":96183635968,"internally_reserved":0,"allocated":135401472,"data_stored":135153287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.219},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.29499999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:02:33 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7569999999999999,"15min":1.0109999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":10.525,"5min":13.773,"15min":13.773},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.522,"15min":0.81299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":5.2859999999999996,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":133750784,"data_stored":133747305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73904128,"data_stored":73851350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":166948864,"data_stored":166941313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:38.974 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:03:38.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'17961", 'reported_seq': 18036, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:33.954129+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:03:33.954129+0000', 'last_peered': '2024-09-18T12:03:33.954129+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:03:33.954129+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'7900", 'ondisk_log_start': "39'7900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6451200, 'num_objects': 1575, 'num_object_clones': 0, 'num_object_copies': 3150, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1575, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17961, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:38.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:03:38.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'18004", 'reported_seq': 18194, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:33.832298+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:03:33.832298+0000', 'last_peered': '2024-09-18T12:03:33.832298+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:03:33.832298+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'8000", 'ondisk_log_start': "39'8000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6361112, 'num_objects': 1554, 'num_object_clones': 0, 'num_object_copies': 3108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1554, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14862, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1982}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:38.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:03:38.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'17865", 'reported_seq': 17950, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:33.899864+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:03:33.899864+0000', 'last_peered': '2024-09-18T12:03:33.899864+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:03:33.899864+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'7800", 'ondisk_log_start': "39'7800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6382802, 'num_objects': 1560, 'num_object_clones': 0, 'num_object_copies': 3120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17863, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1967}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:38.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:03:38.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'17834", 'reported_seq': 17925, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:34.010480+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:03:34.010480+0000', 'last_peered': '2024-09-18T12:03:34.010480+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:03:34.010480+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'7800", 'ondisk_log_start': "39'7800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6300882, 'num_objects': 1540, 'num_object_clones': 0, 'num_object_copies': 3080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17832, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2019}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:38.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:03:38.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'17879", 'reported_seq': 21215, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:34.002279+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:03:34.002279+0000', 'last_peered': '2024-09-18T12:03:34.002279+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:03:34.002279+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'7800", 'ondisk_log_start': "39'7800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6419666, 'num_objects': 1569, 'num_object_clones': 0, 'num_object_copies': 3138, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1907, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1569, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17877, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1629, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -338}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:38.975 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:03:38.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'17913", 'reported_seq': 17973, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:33.844553+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:03:33.844553+0000', 'last_peered': '2024-09-18T12:03:33.844553+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:03:33.844553+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'7900", 'ondisk_log_start': "39'7900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6350034, 'num_objects': 1552, 'num_object_clones': 0, 'num_object_copies': 3104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17911, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1998}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:38.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:03:38.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'17903", 'reported_seq': 25316, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:03:33.798936+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:03:33.798936+0000', 'last_peered': '2024-09-18T12:03:33.798936+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:03:33.798936+0000', 'last_undegraded': '2024-09-18T12:03:33.798936+0000', 'last_fullsized': '2024-09-18T12:03:33.798936+0000', 'mapping_epoch': 42, 'log_start': "39'7900", 'ondisk_log_start': "39'7900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6533120, 'num_objects': 1595, 'num_object_clones': 0, 'num_object_copies': 3190, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1595, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14383, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1595}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:38.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:03:38.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'17921", 'reported_seq': 18007, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:33.937175+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:03:33.937175+0000', 'last_peered': '2024-09-18T12:03:33.937175+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:03:33.937175+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'7900", 'ondisk_log_start': "39'7900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6582889, 'num_objects': 1608, 'num_object_clones': 0, 'num_object_copies': 3216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17920, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1927}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:38.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:03:38.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'17685", 'reported_seq': 17770, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:33.952370+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:03:33.952370+0000', 'last_peered': '2024-09-18T12:03:33.952370+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:03:33.952370+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'7600", 'ondisk_log_start': "39'7600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6198482, 'num_objects': 1515, 'num_object_clones': 0, 'num_object_copies': 3030, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17683, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2078}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:38.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:03:38.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'17987", 'reported_seq': 18076, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:33.971573+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:03:33.971573+0000', 'last_peered': '2024-09-18T12:03:33.971573+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:03:33.971573+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'7900", 'ondisk_log_start': "39'7900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6248868, 'num_objects': 1529, 'num_object_clones': 0, 'num_object_copies': 3058, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17984, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1994}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:38.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:03:39.870 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:39.871 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30253/45536 objects degraded (66.438%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":619,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22768,"data_bytes":93175903,"bytes_used":1297416192,"bytes_avail":288612876288,"bytes_total":289910292480,"degraded_objects":30253,"degraded_total":45536,"degraded_ratio":0.66437543921293041,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":251,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":523},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:03:39.900 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:03:39.901 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:40.583 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:40.609 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:40.659 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-09-18T12:03:39.247423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93175903,"num_objects":22768,"num_object_clones":0,"num_object_copies":45536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22768,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272451,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12341,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160659,"ondisk_log_size":160659,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1267008,"kb_used_data":356840,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281848512,"statfs":{"total":289910292480,"available":288612876288,"internally_reserved":0,"allocated":365404160,"data_stored":364609856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,10,6,15,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25751552,"num_objects":-6287,"num_object_clones":0,"num_object_copies":-12574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002618"},"pg_stats":[{"pgid":"2.f","version":"43'18099","reported_seq":18174,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.952790+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:38.952790+0000","last_peered":"2024-09-18T12:03:38.952790+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:38.952790+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5885952,"num_objects":1437,"num_object_clones":0,"num_object_copies":2874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18099,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2162}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18015","reported_seq":18177,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:38.836261+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:38.836261+0000","last_peered":"2024-09-18T12:03:38.836261+0000","last_clean":"2024-09-18T12:03:38.836261+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:38.836261+0000","last_undegraded":"2024-09-18T12:03:38.836261+0000","last_fullsized":"2024-09-18T12:03:38.836261+0000","mapping_epoch":29,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5804032,"num_objects":1417,"num_object_clones":0,"num_object_copies":2834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15207,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18160","reported_seq":18350,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.731180+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:38.731180+0000","last_peered":"2024-09-18T12:03:38.731180+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:38.731180+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5722136,"num_objects":1398,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15018,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2138}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17869","reported_seq":17902,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:38.811384+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:38.811384+0000","last_peered":"2024-09-18T12:03:38.811384+0000","last_clean":"2024-09-18T12:03:38.811384+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:38.811384+0000","last_undegraded":"2024-09-18T12:03:38.811384+0000","last_fullsized":"2024-09-18T12:03:38.811384+0000","mapping_epoch":29,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5697536,"num_objects":1391,"num_object_clones":0,"num_object_copies":2782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17869,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18072","reported_seq":18267,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:38.914121+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:38.914121+0000","last_peered":"2024-09-18T12:03:38.914121+0000","last_clean":"2024-09-18T12:03:38.914121+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:38.914121+0000","last_undegraded":"2024-09-18T12:03:38.914121+0000","last_fullsized":"2024-09-18T12:03:38.914121+0000","mapping_epoch":29,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5632000,"num_objects":1375,"num_object_clones":0,"num_object_copies":2750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15237,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18006","reported_seq":18088,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.940800+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:38.940800+0000","last_peered":"2024-09-18T12:03:38.940800+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:38.940800+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5805266,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18004,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17945","reported_seq":18036,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.990208+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:38.990208+0000","last_peered":"2024-09-18T12:03:38.990208+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:38.990208+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5846226,"num_objects":1429,"num_object_clones":0,"num_object_copies":2858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17943,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2130}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18218","reported_seq":18447,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:39.036286+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:39.036286+0000","last_peered":"2024-09-18T12:03:39.036286+0000","last_clean":"2024-09-18T12:03:39.036286+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:39.036286+0000","last_undegraded":"2024-09-18T12:03:39.036286+0000","last_fullsized":"2024-09-18T12:03:39.036286+0000","mapping_epoch":29,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5668864,"num_objects":1384,"num_object_clones":0,"num_object_copies":2768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14535,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'17902","reported_seq":25034,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:35.728371+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:35.728371+0000","last_peered":"2024-09-18T12:03:35.728371+0000","last_clean":"2024-09-18T12:03:35.728371+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:35.728371+0000","last_undegraded":"2024-09-18T12:03:35.728371+0000","last_fullsized":"2024-09-18T12:03:35.728371+0000","mapping_epoch":38,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6157522,"num_objects":1505,"num_object_clones":0,"num_object_copies":3010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17900,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18017","reported_seq":21527,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:39.018646+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:39.018646+0000","last_peered":"2024-09-18T12:03:39.018646+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:39.018646+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5854418,"num_objects":1431,"num_object_clones":0,"num_object_copies":2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1818,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18015,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1718,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-387}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18055","reported_seq":18115,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.830858+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:38.830858+0000","last_peered":"2024-09-18T12:03:38.830858+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:38.830858+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5768402,"num_objects":1410,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18053,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18052","reported_seq":25465,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:38.880298+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:38.880298+0000","last_peered":"2024-09-18T12:03:38.880298+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:38.880298+0000","last_undegraded":"2024-09-18T12:03:38.880298+0000","last_fullsized":"2024-09-18T12:03:38.880298+0000","mapping_epoch":42,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5922816,"num_objects":1446,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14532,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1446}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18058","reported_seq":18144,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.974231+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:38.974231+0000","last_peered":"2024-09-18T12:03:38.974231+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:38.974231+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6021737,"num_objects":1471,"num_object_clones":0,"num_object_copies":2942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18057,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18042","reported_seq":25192,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:35.823213+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:35.823213+0000","last_peered":"2024-09-18T12:03:35.823213+0000","last_clean":"2024-09-18T12:03:35.823213+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:35.823213+0000","last_undegraded":"2024-09-18T12:03:35.823213+0000","last_fullsized":"2024-09-18T12:03:35.823213+0000","mapping_epoch":40,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6063931,"num_objects":1483,"num_object_clones":0,"num_object_copies":2966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18039,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17813","reported_seq":17898,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.928531+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:38.928531+0000","last_peered":"2024-09-18T12:03:38.928531+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:38.928531+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5674194,"num_objects":1387,"num_object_clones":0,"num_object_copies":2774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17811,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18133","reported_seq":18222,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.964363+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:38.964363+0000","last_peered":"2024-09-18T12:03:38.964363+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:38.964363+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5650852,"num_objects":1383,"num_object_clones":0,"num_object_copies":2766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18130,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":93175884,"num_objects":22766,"num_object_clones":0,"num_object_copies":45532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22766,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272449,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12340,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":363417600,"data_stored":363353792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160656,"ondisk_log_size":160656,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051701,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457220,"kb_used_data":159356,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93914620,"statfs":{"total":96636764160,"available":96168570880,"internally_reserved":0,"allocated":163180544,"data_stored":162931909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.27100000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.872}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":16.847999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.40699999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855463,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383212,"kb_used_data":72868,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93988628,"statfs":{"total":96636764160,"available":96244355072,"internally_reserved":0,"allocated":74616832,"data_stored":74319348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,3,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.26300000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.23300000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426576,"kb_used_data":124616,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93945264,"statfs":{"total":96636764160,"available":96199950336,"internally_reserved":0,"allocated":127606784,"data_stored":127358599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,6,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.29899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.27200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":127176704,"data_stored":127173225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73904128,"data_stored":73851350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162336768,"data_stored":162329217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:43.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:44.364 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:44.382 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:44.396 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-18T12:03:43.248357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92151903,"num_objects":22518,"num_object_clones":0,"num_object_copies":45036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22518,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272701,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12341,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160709,"ondisk_log_size":160709,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1252444,"kb_used_data":350592,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281863076,"statfs":{"total":289910292480,"available":288627789824,"internally_reserved":0,"allocated":359006208,"data_stored":358211904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,6,10,7,10,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17858560,"num_objects":-4360,"num_object_clones":0,"num_object_copies":-8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002719"},"pg_stats":[{"pgid":"2.f","version":"43'18099","reported_seq":18174,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.952790+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:38.952790+0000","last_peered":"2024-09-18T12:03:38.952790+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:38.952790+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5885952,"num_objects":1437,"num_object_clones":0,"num_object_copies":2874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18099,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2162}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18015","reported_seq":18177,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:38.836261+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:38.836261+0000","last_peered":"2024-09-18T12:03:38.836261+0000","last_clean":"2024-09-18T12:03:38.836261+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:38.836261+0000","last_undegraded":"2024-09-18T12:03:38.836261+0000","last_fullsized":"2024-09-18T12:03:38.836261+0000","mapping_epoch":29,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5804032,"num_objects":1417,"num_object_clones":0,"num_object_copies":2834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15207,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18160","reported_seq":18350,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.731180+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:38.731180+0000","last_peered":"2024-09-18T12:03:38.731180+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:38.731180+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5722136,"num_objects":1398,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15018,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2138}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17869","reported_seq":17902,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:38.811384+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:38.811384+0000","last_peered":"2024-09-18T12:03:38.811384+0000","last_clean":"2024-09-18T12:03:38.811384+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:38.811384+0000","last_undegraded":"2024-09-18T12:03:38.811384+0000","last_fullsized":"2024-09-18T12:03:38.811384+0000","mapping_epoch":29,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5697536,"num_objects":1391,"num_object_clones":0,"num_object_copies":2782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17869,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18072","reported_seq":18267,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:38.914121+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:38.914121+0000","last_peered":"2024-09-18T12:03:38.914121+0000","last_clean":"2024-09-18T12:03:38.914121+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:38.914121+0000","last_undegraded":"2024-09-18T12:03:38.914121+0000","last_fullsized":"2024-09-18T12:03:38.914121+0000","mapping_epoch":29,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5632000,"num_objects":1375,"num_object_clones":0,"num_object_copies":2750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15237,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18006","reported_seq":18088,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.940800+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:38.940800+0000","last_peered":"2024-09-18T12:03:38.940800+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:38.940800+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5805266,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18004,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17945","reported_seq":18036,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.990208+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:38.990208+0000","last_peered":"2024-09-18T12:03:38.990208+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:38.990208+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5846226,"num_objects":1429,"num_object_clones":0,"num_object_copies":2858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17943,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2130}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18218","reported_seq":18447,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:39.036286+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:39.036286+0000","last_peered":"2024-09-18T12:03:39.036286+0000","last_clean":"2024-09-18T12:03:39.036286+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:39.036286+0000","last_undegraded":"2024-09-18T12:03:39.036286+0000","last_fullsized":"2024-09-18T12:03:39.036286+0000","mapping_epoch":29,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5668864,"num_objects":1384,"num_object_clones":0,"num_object_copies":2768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14535,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18026","reported_seq":25158,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:40.922030+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:40.922030+0000","last_peered":"2024-09-18T12:03:40.922030+0000","last_clean":"2024-09-18T12:03:40.922030+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:40.922030+0000","last_undegraded":"2024-09-18T12:03:40.922030+0000","last_fullsized":"2024-09-18T12:03:40.922030+0000","mapping_epoch":38,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5649618,"num_objects":1381,"num_object_clones":0,"num_object_copies":2762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18024,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18017","reported_seq":21527,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:39.018646+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:39.018646+0000","last_peered":"2024-09-18T12:03:39.018646+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:39.018646+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5854418,"num_objects":1431,"num_object_clones":0,"num_object_copies":2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1818,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18015,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1718,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-387}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18055","reported_seq":18115,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.830858+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:38.830858+0000","last_peered":"2024-09-18T12:03:38.830858+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:38.830858+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5768402,"num_objects":1410,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18053,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18052","reported_seq":25465,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:38.880298+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:38.880298+0000","last_peered":"2024-09-18T12:03:38.880298+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:38.880298+0000","last_undegraded":"2024-09-18T12:03:38.880298+0000","last_fullsized":"2024-09-18T12:03:38.880298+0000","mapping_epoch":42,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5922816,"num_objects":1446,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14532,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1446}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18058","reported_seq":18144,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.974231+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:38.974231+0000","last_peered":"2024-09-18T12:03:38.974231+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:38.974231+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6021737,"num_objects":1471,"num_object_clones":0,"num_object_copies":2942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18057,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18168","reported_seq":25318,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:40.918238+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:40.918238+0000","last_peered":"2024-09-18T12:03:40.918238+0000","last_clean":"2024-09-18T12:03:40.918238+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:40.918238+0000","last_undegraded":"2024-09-18T12:03:40.918238+0000","last_fullsized":"2024-09-18T12:03:40.918238+0000","mapping_epoch":40,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5547835,"num_objects":1357,"num_object_clones":0,"num_object_copies":2714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18165,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17813","reported_seq":17898,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.928531+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:38.928531+0000","last_peered":"2024-09-18T12:03:38.928531+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:38.928531+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5674194,"num_objects":1387,"num_object_clones":0,"num_object_copies":2774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17811,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18133","reported_seq":18222,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.964363+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:38.964363+0000","last_peered":"2024-09-18T12:03:38.964363+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:38.964363+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5650852,"num_objects":1383,"num_object_clones":0,"num_object_copies":2766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18130,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":92151884,"num_objects":22516,"num_object_clones":0,"num_object_copies":45032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272699,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12340,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356823040,"data_stored":356759232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160706,"ondisk_log_size":160706,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051701,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457220,"kb_used_data":159356,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93914620,"statfs":{"total":96636764160,"available":96168570880,"internally_reserved":0,"allocated":163180544,"data_stored":162931909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.27100000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.872}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":16.847999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.40699999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855464,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368648,"kb_used_data":66620,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94003192,"statfs":{"total":96636764160,"available":96259268608,"internally_reserved":0,"allocated":68218880,"data_stored":67921396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.216}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.64600000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426576,"kb_used_data":124616,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93945264,"statfs":{"total":96636764160,"available":96199950336,"internally_reserved":0,"allocated":127606784,"data_stored":127358599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,6,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.29899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.27200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":127176704,"data_stored":127173225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67309568,"data_stored":67256790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162336768,"data_stored":162329217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:44.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:44.956 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:44.963 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:44.980 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-18T12:03:43.248357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92151903,"num_objects":22518,"num_object_clones":0,"num_object_copies":45036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22518,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272701,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12341,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160709,"ondisk_log_size":160709,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1252444,"kb_used_data":350592,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281863076,"statfs":{"total":289910292480,"available":288627789824,"internally_reserved":0,"allocated":359006208,"data_stored":358211904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,6,10,7,10,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17858560,"num_objects":-4360,"num_object_clones":0,"num_object_copies":-8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002719"},"pg_stats":[{"pgid":"2.f","version":"43'18099","reported_seq":18174,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.952790+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:38.952790+0000","last_peered":"2024-09-18T12:03:38.952790+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:38.952790+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5885952,"num_objects":1437,"num_object_clones":0,"num_object_copies":2874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18099,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2162}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18015","reported_seq":18177,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:38.836261+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:38.836261+0000","last_peered":"2024-09-18T12:03:38.836261+0000","last_clean":"2024-09-18T12:03:38.836261+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:38.836261+0000","last_undegraded":"2024-09-18T12:03:38.836261+0000","last_fullsized":"2024-09-18T12:03:38.836261+0000","mapping_epoch":29,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5804032,"num_objects":1417,"num_object_clones":0,"num_object_copies":2834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15207,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18160","reported_seq":18350,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.731180+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:38.731180+0000","last_peered":"2024-09-18T12:03:38.731180+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:38.731180+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5722136,"num_objects":1398,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15018,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2138}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'17869","reported_seq":17902,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:38.811384+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:38.811384+0000","last_peered":"2024-09-18T12:03:38.811384+0000","last_clean":"2024-09-18T12:03:38.811384+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:38.811384+0000","last_undegraded":"2024-09-18T12:03:38.811384+0000","last_fullsized":"2024-09-18T12:03:38.811384+0000","mapping_epoch":29,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5697536,"num_objects":1391,"num_object_clones":0,"num_object_copies":2782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17869,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18072","reported_seq":18267,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:38.914121+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:38.914121+0000","last_peered":"2024-09-18T12:03:38.914121+0000","last_clean":"2024-09-18T12:03:38.914121+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:38.914121+0000","last_undegraded":"2024-09-18T12:03:38.914121+0000","last_fullsized":"2024-09-18T12:03:38.914121+0000","mapping_epoch":29,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5632000,"num_objects":1375,"num_object_clones":0,"num_object_copies":2750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15237,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18006","reported_seq":18088,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.940800+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:38.940800+0000","last_peered":"2024-09-18T12:03:38.940800+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:38.940800+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5805266,"num_objects":1419,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18004,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2108}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'17945","reported_seq":18036,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.990208+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:38.990208+0000","last_peered":"2024-09-18T12:03:38.990208+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:38.990208+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5846226,"num_objects":1429,"num_object_clones":0,"num_object_copies":2858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17943,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2130}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18218","reported_seq":18447,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:39.036286+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:39.036286+0000","last_peered":"2024-09-18T12:03:39.036286+0000","last_clean":"2024-09-18T12:03:39.036286+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:39.036286+0000","last_undegraded":"2024-09-18T12:03:39.036286+0000","last_fullsized":"2024-09-18T12:03:39.036286+0000","mapping_epoch":29,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5668864,"num_objects":1384,"num_object_clones":0,"num_object_copies":2768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14535,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18026","reported_seq":25158,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:40.922030+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:40.922030+0000","last_peered":"2024-09-18T12:03:40.922030+0000","last_clean":"2024-09-18T12:03:40.922030+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:40.922030+0000","last_undegraded":"2024-09-18T12:03:40.922030+0000","last_fullsized":"2024-09-18T12:03:40.922030+0000","mapping_epoch":38,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5649618,"num_objects":1381,"num_object_clones":0,"num_object_copies":2762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18024,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18017","reported_seq":21527,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:39.018646+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:39.018646+0000","last_peered":"2024-09-18T12:03:39.018646+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:39.018646+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5854418,"num_objects":1431,"num_object_clones":0,"num_object_copies":2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1818,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18015,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1718,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-387}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18055","reported_seq":18115,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.830858+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:38.830858+0000","last_peered":"2024-09-18T12:03:38.830858+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:38.830858+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5768402,"num_objects":1410,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18053,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18052","reported_seq":25465,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:38.880298+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:38.880298+0000","last_peered":"2024-09-18T12:03:38.880298+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:38.880298+0000","last_undegraded":"2024-09-18T12:03:38.880298+0000","last_fullsized":"2024-09-18T12:03:38.880298+0000","mapping_epoch":42,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5922816,"num_objects":1446,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14532,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1446}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18058","reported_seq":18144,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.974231+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:38.974231+0000","last_peered":"2024-09-18T12:03:38.974231+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:38.974231+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6021737,"num_objects":1471,"num_object_clones":0,"num_object_copies":2942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18057,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18168","reported_seq":25318,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:40.918238+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:40.918238+0000","last_peered":"2024-09-18T12:03:40.918238+0000","last_clean":"2024-09-18T12:03:40.918238+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:40.918238+0000","last_undegraded":"2024-09-18T12:03:40.918238+0000","last_fullsized":"2024-09-18T12:03:40.918238+0000","mapping_epoch":40,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5547835,"num_objects":1357,"num_object_clones":0,"num_object_copies":2714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18165,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17813","reported_seq":17898,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.928531+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:38.928531+0000","last_peered":"2024-09-18T12:03:38.928531+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:38.928531+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'7800","ondisk_log_start":"39'7800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5674194,"num_objects":1387,"num_object_clones":0,"num_object_copies":2774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17811,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18133","reported_seq":18222,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:38.964363+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:38.964363+0000","last_peered":"2024-09-18T12:03:38.964363+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:38.964363+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5650852,"num_objects":1383,"num_object_clones":0,"num_object_copies":2766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18130,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":92151884,"num_objects":22516,"num_object_clones":0,"num_object_copies":45032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272699,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12340,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356823040,"data_stored":356759232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160706,"ondisk_log_size":160706,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051701,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457220,"kb_used_data":159356,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93914620,"statfs":{"total":96636764160,"available":96168570880,"internally_reserved":0,"allocated":163180544,"data_stored":162931909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.27100000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.872}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":16.847999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.40699999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855464,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368648,"kb_used_data":66620,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94003192,"statfs":{"total":96636764160,"available":96259268608,"internally_reserved":0,"allocated":68218880,"data_stored":67921396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.216}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.64600000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426576,"kb_used_data":124616,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93945264,"statfs":{"total":96636764160,"available":96199950336,"internally_reserved":0,"allocated":127606784,"data_stored":127358599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,6,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.29899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.27200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":127176704,"data_stored":127173225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67309568,"data_stored":67256790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162336768,"data_stored":162329217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:44.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:03:44.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'18099", 'reported_seq': 18174, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:38.952790+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:03:38.952790+0000', 'last_peered': '2024-09-18T12:03:38.952790+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:03:38.952790+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'8000", 'ondisk_log_start': "39'8000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5885952, 'num_objects': 1437, 'num_object_clones': 0, 'num_object_copies': 2874, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18099, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2162}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:44.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:03:44.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'18160", 'reported_seq': 18350, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:38.731180+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:03:38.731180+0000', 'last_peered': '2024-09-18T12:03:38.731180+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:03:38.731180+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'8100", 'ondisk_log_start': "39'8100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5722136, 'num_objects': 1398, 'num_object_clones': 0, 'num_object_copies': 2796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1398, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15018, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2138}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:44.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:03:44.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'18006", 'reported_seq': 18088, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:38.940800+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:03:38.940800+0000', 'last_peered': '2024-09-18T12:03:38.940800+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:03:38.940800+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'8000", 'ondisk_log_start': "39'8000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5805266, 'num_objects': 1419, 'num_object_clones': 0, 'num_object_copies': 2838, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18004, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2108}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:44.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:03:44.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'17945", 'reported_seq': 18036, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:38.990208+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:03:38.990208+0000', 'last_peered': '2024-09-18T12:03:38.990208+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:03:38.990208+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'7900", 'ondisk_log_start': "39'7900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5846226, 'num_objects': 1429, 'num_object_clones': 0, 'num_object_copies': 2858, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1429, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17943, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2130}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:44.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:03:44.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'18017", 'reported_seq': 21527, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:39.018646+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:03:39.018646+0000', 'last_peered': '2024-09-18T12:03:39.018646+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:03:39.018646+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'8000", 'ondisk_log_start': "39'8000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5854418, 'num_objects': 1431, 'num_object_clones': 0, 'num_object_copies': 2862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1818, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18015, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1718, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -387}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:44.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:03:44.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'18055", 'reported_seq': 18115, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:38.830858+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:03:38.830858+0000', 'last_peered': '2024-09-18T12:03:38.830858+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:03:38.830858+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'8000", 'ondisk_log_start': "39'8000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5768402, 'num_objects': 1410, 'num_object_clones': 0, 'num_object_copies': 2820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18053, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2140}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:44.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:03:44.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'18052", 'reported_seq': 25465, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:03:38.880298+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:03:38.880298+0000', 'last_peered': '2024-09-18T12:03:38.880298+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:03:38.880298+0000', 'last_undegraded': '2024-09-18T12:03:38.880298+0000', 'last_fullsized': '2024-09-18T12:03:38.880298+0000', 'mapping_epoch': 42, 'log_start': "39'8000", 'ondisk_log_start': "39'8000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5922816, 'num_objects': 1446, 'num_object_clones': 0, 'num_object_copies': 2892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1446, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14532, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1446}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:44.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:03:44.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'18058", 'reported_seq': 18144, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:38.974231+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:03:38.974231+0000', 'last_peered': '2024-09-18T12:03:38.974231+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:03:38.974231+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'8000", 'ondisk_log_start': "39'8000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6021737, 'num_objects': 1471, 'num_object_clones': 0, 'num_object_copies': 2942, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1471, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18057, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2064}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:44.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:03:44.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'17813", 'reported_seq': 17898, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:38.928531+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:03:38.928531+0000', 'last_peered': '2024-09-18T12:03:38.928531+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:03:38.928531+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'7800", 'ondisk_log_start': "39'7800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5674194, 'num_objects': 1387, 'num_object_clones': 0, 'num_object_copies': 2774, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17811, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2206}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:44.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:03:44.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'18133", 'reported_seq': 18222, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:38.964363+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:03:38.964363+0000', 'last_peered': '2024-09-18T12:03:38.964363+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:03:38.964363+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'8100", 'ondisk_log_start': "39'8100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5650852, 'num_objects': 1383, 'num_object_clones': 0, 'num_object_copies': 2766, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18130, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:44.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:03:45.650 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:45.651 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30253/45036 objects degraded (67.175%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":625,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22518,"data_bytes":92151903,"bytes_used":1282502656,"bytes_avail":288627789824,"bytes_total":289910292480,"degraded_objects":30253,"degraded_total":45036,"degraded_ratio":0.67175148769872994,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":363},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:03:45.709 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:03:45.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:46.387 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:46.433 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:46.434 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-18T12:03:45.248762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":84222047,"num_objects":20582,"num_object_clones":0,"num_object_copies":41164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":274637,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12436,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1249668,"kb_used_data":339488,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281865852,"statfs":{"total":289910292480,"available":288630632448,"internally_reserved":0,"allocated":347635712,"data_stored":346841408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,6,6,9,16,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25788416,"num_objects":-6296,"num_object_clones":0,"num_object_copies":-12592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6296,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002668"},"pg_stats":[{"pgid":"2.f","version":"43'18242","reported_seq":18315,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:44.045576+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:44.045576+0000","last_peered":"2024-09-18T12:03:44.045576+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:44.045576+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5300224,"num_objects":1294,"num_object_clones":0,"num_object_copies":2588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18242,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2305}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18163","reported_seq":18325,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:43.874034+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:43.874034+0000","last_peered":"2024-09-18T12:03:43.874034+0000","last_clean":"2024-09-18T12:03:43.874034+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:43.874034+0000","last_undegraded":"2024-09-18T12:03:43.874034+0000","last_fullsized":"2024-09-18T12:03:43.874034+0000","mapping_epoch":29,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5197824,"num_objects":1269,"num_object_clones":0,"num_object_copies":2538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15355,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18300","reported_seq":18490,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:43.868626+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:43.868626+0000","last_peered":"2024-09-18T12:03:43.868626+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:43.868626+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5148696,"num_objects":1258,"num_object_clones":0,"num_object_copies":2516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1258,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15158,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2278}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18008","reported_seq":18041,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:43.896694+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:43.896694+0000","last_peered":"2024-09-18T12:03:43.896694+0000","last_clean":"2024-09-18T12:03:43.896694+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:43.896694+0000","last_undegraded":"2024-09-18T12:03:43.896694+0000","last_fullsized":"2024-09-18T12:03:43.896694+0000","mapping_epoch":29,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5128192,"num_objects":1252,"num_object_clones":0,"num_object_copies":2504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18008,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18209","reported_seq":18404,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:43.973209+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:43.973209+0000","last_peered":"2024-09-18T12:03:43.973209+0000","last_clean":"2024-09-18T12:03:43.973209+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:43.973209+0000","last_undegraded":"2024-09-18T12:03:43.973209+0000","last_fullsized":"2024-09-18T12:03:43.973209+0000","mapping_epoch":29,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5070848,"num_objects":1238,"num_object_clones":0,"num_object_copies":2476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15374,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18144","reported_seq":18229,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:43.937977+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:43.937977+0000","last_peered":"2024-09-18T12:03:43.937977+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:43.937977+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5240018,"num_objects":1281,"num_object_clones":0,"num_object_copies":2562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18142,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18087","reported_seq":18178,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:44.001561+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:44.001561+0000","last_peered":"2024-09-18T12:03:44.001561+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:44.001561+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5264594,"num_objects":1287,"num_object_clones":0,"num_object_copies":2574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18085,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18361","reported_seq":18590,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:44.051323+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:44.051323+0000","last_peered":"2024-09-18T12:03:44.051323+0000","last_clean":"2024-09-18T12:03:44.051323+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:44.051323+0000","last_undegraded":"2024-09-18T12:03:44.051323+0000","last_fullsized":"2024-09-18T12:03:44.051323+0000","mapping_epoch":29,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5083136,"num_objects":1241,"num_object_clones":0,"num_object_copies":2482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18026","reported_seq":25158,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:40.922030+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:40.922030+0000","last_peered":"2024-09-18T12:03:40.922030+0000","last_clean":"2024-09-18T12:03:40.922030+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:40.922030+0000","last_undegraded":"2024-09-18T12:03:40.922030+0000","last_fullsized":"2024-09-18T12:03:40.922030+0000","mapping_epoch":38,"log_start":"39'8000","ondisk_log_start":"39'8000","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5649618,"num_objects":1381,"num_object_clones":0,"num_object_copies":2762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18024,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18145","reported_seq":21849,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:44.071517+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:44.071517+0000","last_peered":"2024-09-18T12:03:44.071517+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:44.071517+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5330130,"num_objects":1303,"num_object_clones":0,"num_object_copies":2606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18143,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1813,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18183","reported_seq":18243,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:43.907705+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:43.907705+0000","last_peered":"2024-09-18T12:03:43.907705+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:43.907705+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5244114,"num_objects":1282,"num_object_clones":0,"num_object_copies":2564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18181,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2268}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18219","reported_seq":25632,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:43.931708+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:43.931708+0000","last_peered":"2024-09-18T12:03:43.931708+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:43.931708+0000","last_undegraded":"2024-09-18T12:03:43.931708+0000","last_fullsized":"2024-09-18T12:03:43.931708+0000","mapping_epoch":42,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5238784,"num_objects":1279,"num_object_clones":0,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14699,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1279}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18179","reported_seq":18265,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:43.878217+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:43.878217+0000","last_peered":"2024-09-18T12:03:43.878217+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:43.878217+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5526121,"num_objects":1350,"num_object_clones":0,"num_object_copies":2700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18178,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2185}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18168","reported_seq":25318,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:40.918238+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:40.918238+0000","last_peered":"2024-09-18T12:03:40.918238+0000","last_clean":"2024-09-18T12:03:40.918238+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:40.918238+0000","last_undegraded":"2024-09-18T12:03:40.918238+0000","last_fullsized":"2024-09-18T12:03:40.918238+0000","mapping_epoch":40,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5547835,"num_objects":1357,"num_object_clones":0,"num_object_copies":2714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18165,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'17955","reported_seq":18040,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:44.032719+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:44.032719+0000","last_peered":"2024-09-18T12:03:44.032719+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:44.032719+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'7900","ondisk_log_start":"39'7900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5092562,"num_objects":1245,"num_object_clones":0,"num_object_copies":2490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17953,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18253","reported_seq":18342,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:43.874851+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:43.874851+0000","last_peered":"2024-09-18T12:03:43.874851+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:43.874851+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5159332,"num_objects":1263,"num_object_clones":0,"num_object_copies":2526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18250,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2260}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":84222028,"num_objects":20580,"num_object_clones":0,"num_object_copies":41160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20580,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":274635,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12435,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":345493504,"data_stored":345429696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051702,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456824,"kb_used_data":154800,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93915016,"statfs":{"total":96636764160,"available":96168976384,"internally_reserved":0,"allocated":158515200,"data_stored":158266565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":5.6840000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":5.8029999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":16.350000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":5.4039999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855464,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368648,"kb_used_data":66620,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94003192,"statfs":{"total":96636764160,"available":96259268608,"internally_reserved":0,"allocated":68218880,"data_stored":67921396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.216}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.64600000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":424196,"kb_used_data":118068,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93947644,"statfs":{"total":96636764160,"available":96202387456,"internally_reserved":0,"allocated":120901632,"data_stored":120653447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":5.0880000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.34000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":120442880,"data_stored":120439401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67309568,"data_stored":67256790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157741056,"data_stored":157733505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:49.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:50.111 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:50.125 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:50.150 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-18T12:03:49.249661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":75247711,"num_objects":18391,"num_object_clones":0,"num_object_copies":36782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":276828,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12526,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160936,"ondisk_log_size":160936,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1232432,"kb_used_data":322256,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281883088,"statfs":{"total":289910292480,"available":288648282112,"internally_reserved":0,"allocated":329990144,"data_stored":329195840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,2,7,9,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25829376,"num_objects":-6306,"num_object_clones":0,"num_object_copies":-12612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6306,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002696"},"pg_stats":[{"pgid":"2.f","version":"43'18394","reported_seq":18469,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.077754+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:49.077754+0000","last_peered":"2024-09-18T12:03:49.077754+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:49.077754+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4677632,"num_objects":1142,"num_object_clones":0,"num_object_copies":2284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18394,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2457}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18296","reported_seq":18458,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:48.895397+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:48.895397+0000","last_peered":"2024-09-18T12:03:48.895397+0000","last_clean":"2024-09-18T12:03:48.895397+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:48.895397+0000","last_undegraded":"2024-09-18T12:03:48.895397+0000","last_fullsized":"2024-09-18T12:03:48.895397+0000","mapping_epoch":29,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":1136,"num_object_clones":0,"num_object_copies":2272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15488,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18451","reported_seq":18641,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:48.876051+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:48.876051+0000","last_peered":"2024-09-18T12:03:48.876051+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:48.876051+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4530200,"num_objects":1107,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15309,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2429}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18139","reported_seq":18170,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:48.929508+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:48.929508+0000","last_peered":"2024-09-18T12:03:48.929508+0000","last_clean":"2024-09-18T12:03:48.929508+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:48.929508+0000","last_undegraded":"2024-09-18T12:03:48.929508+0000","last_fullsized":"2024-09-18T12:03:48.929508+0000","mapping_epoch":29,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4591616,"num_objects":1121,"num_object_clones":0,"num_object_copies":2242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18139,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18328","reported_seq":18523,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:49.086476+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:49.086476+0000","last_peered":"2024-09-18T12:03:49.086476+0000","last_clean":"2024-09-18T12:03:49.086476+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:49.086476+0000","last_undegraded":"2024-09-18T12:03:49.086476+0000","last_fullsized":"2024-09-18T12:03:49.086476+0000","mapping_epoch":29,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4583424,"num_objects":1119,"num_object_clones":0,"num_object_copies":2238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15493,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18274","reported_seq":18359,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.085459+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:49.085459+0000","last_peered":"2024-09-18T12:03:49.085459+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:49.085459+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4707538,"num_objects":1151,"num_object_clones":0,"num_object_copies":2302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18272,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18219","reported_seq":18310,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.058635+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:49.058635+0000","last_peered":"2024-09-18T12:03:49.058635+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:49.058635+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4723922,"num_objects":1155,"num_object_clones":0,"num_object_copies":2310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18217,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18476","reported_seq":18704,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:49.096593+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:49.096593+0000","last_peered":"2024-09-18T12:03:49.096593+0000","last_clean":"2024-09-18T12:03:49.096593+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:49.096593+0000","last_undegraded":"2024-09-18T12:03:49.096593+0000","last_fullsized":"2024-09-18T12:03:49.096593+0000","mapping_epoch":29,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4612096,"num_objects":1126,"num_object_clones":0,"num_object_copies":2252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14793,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18157","reported_seq":25289,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:45.986688+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:45.986688+0000","last_peered":"2024-09-18T12:03:45.986688+0000","last_clean":"2024-09-18T12:03:45.986688+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:45.986688+0000","last_undegraded":"2024-09-18T12:03:45.986688+0000","last_fullsized":"2024-09-18T12:03:45.986688+0000","mapping_epoch":38,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5113042,"num_objects":1250,"num_object_clones":0,"num_object_copies":2500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18155,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18292","reported_seq":22176,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.118523+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:49.118523+0000","last_peered":"2024-09-18T12:03:49.118523+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:49.118523+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4728018,"num_objects":1156,"num_object_clones":0,"num_object_copies":2312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18290,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1903,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-477}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18299","reported_seq":18359,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:48.822593+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:48.822593+0000","last_peered":"2024-09-18T12:03:48.822593+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:48.822593+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4768978,"num_objects":1166,"num_object_clones":0,"num_object_copies":2332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18297,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2384}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18353","reported_seq":25765,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:48.868549+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:48.868549+0000","last_peered":"2024-09-18T12:03:48.868549+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:48.868549+0000","last_undegraded":"2024-09-18T12:03:48.868549+0000","last_fullsized":"2024-09-18T12:03:48.868549+0000","mapping_epoch":42,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4689920,"num_objects":1145,"num_object_clones":0,"num_object_copies":2290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14833,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1145}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18333","reported_seq":18419,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.122755+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:49.122755+0000","last_peered":"2024-09-18T12:03:49.122755+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:49.122755+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4895337,"num_objects":1196,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18332,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18320","reported_seq":25470,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:45.916051+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:45.916051+0000","last_peered":"2024-09-18T12:03:45.916051+0000","last_clean":"2024-09-18T12:03:45.916051+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:45.916051+0000","last_undegraded":"2024-09-18T12:03:45.916051+0000","last_fullsized":"2024-09-18T12:03:45.916051+0000","mapping_epoch":40,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4925243,"num_objects":1205,"num_object_clones":0,"num_object_copies":2410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18317,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18104","reported_seq":18189,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.080353+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:49.080353+0000","last_peered":"2024-09-18T12:03:49.080353+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:49.080353+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4482258,"num_objects":1096,"num_object_clones":0,"num_object_copies":2192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18102,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2497}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18398","reported_seq":18487,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:48.989113+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:48.989113+0000","last_peered":"2024-09-18T12:03:48.989113+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:48.989113+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4565412,"num_objects":1118,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18395,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":75247692,"num_objects":18389,"num_object_clones":0,"num_object_copies":36778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18389,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":276826,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12525,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327647232,"data_stored":327583424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051703,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456920,"kb_used_data":150732,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93914920,"statfs":{"total":96636764160,"available":96168878080,"internally_reserved":0,"allocated":154349568,"data_stored":154100933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.34000000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.187}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.221},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.373}]}]},{"osd":1,"up_from":35,"seq":150323855465,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":366308,"kb_used_data":60120,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94005532,"statfs":{"total":96636764160,"available":96261664768,"internally_reserved":0,"allocated":61562880,"data_stored":61265396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.499}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.34999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409204,"kb_used_data":111404,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93962636,"statfs":{"total":96636764160,"available":96217739264,"internally_reserved":0,"allocated":114077696,"data_stored":113829511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,3,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.74199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113668096,"data_stored":113664617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60428288,"data_stored":60375510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153550848,"data_stored":153543297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:50.151 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:50.838 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:50.842 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:50.855 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-18T12:03:49.249661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":75247711,"num_objects":18391,"num_object_clones":0,"num_object_copies":36782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18391,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":276828,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12526,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160936,"ondisk_log_size":160936,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1232432,"kb_used_data":322256,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":281883088,"statfs":{"total":289910292480,"available":288648282112,"internally_reserved":0,"allocated":329990144,"data_stored":329195840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,2,7,9,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25829376,"num_objects":-6306,"num_object_clones":0,"num_object_copies":-12612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6306,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002696"},"pg_stats":[{"pgid":"2.f","version":"43'18394","reported_seq":18469,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.077754+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:49.077754+0000","last_peered":"2024-09-18T12:03:49.077754+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:49.077754+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4677632,"num_objects":1142,"num_object_clones":0,"num_object_copies":2284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18394,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2457}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18296","reported_seq":18458,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:48.895397+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:48.895397+0000","last_peered":"2024-09-18T12:03:48.895397+0000","last_clean":"2024-09-18T12:03:48.895397+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:48.895397+0000","last_undegraded":"2024-09-18T12:03:48.895397+0000","last_fullsized":"2024-09-18T12:03:48.895397+0000","mapping_epoch":29,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":1136,"num_object_clones":0,"num_object_copies":2272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15488,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18451","reported_seq":18641,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:48.876051+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:48.876051+0000","last_peered":"2024-09-18T12:03:48.876051+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:48.876051+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4530200,"num_objects":1107,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15309,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2429}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18139","reported_seq":18170,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:48.929508+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:48.929508+0000","last_peered":"2024-09-18T12:03:48.929508+0000","last_clean":"2024-09-18T12:03:48.929508+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:48.929508+0000","last_undegraded":"2024-09-18T12:03:48.929508+0000","last_fullsized":"2024-09-18T12:03:48.929508+0000","mapping_epoch":29,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4591616,"num_objects":1121,"num_object_clones":0,"num_object_copies":2242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18139,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18328","reported_seq":18523,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:49.086476+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:49.086476+0000","last_peered":"2024-09-18T12:03:49.086476+0000","last_clean":"2024-09-18T12:03:49.086476+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:49.086476+0000","last_undegraded":"2024-09-18T12:03:49.086476+0000","last_fullsized":"2024-09-18T12:03:49.086476+0000","mapping_epoch":29,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4583424,"num_objects":1119,"num_object_clones":0,"num_object_copies":2238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15493,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18274","reported_seq":18359,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.085459+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:49.085459+0000","last_peered":"2024-09-18T12:03:49.085459+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:49.085459+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4707538,"num_objects":1151,"num_object_clones":0,"num_object_copies":2302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18272,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18219","reported_seq":18310,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.058635+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:49.058635+0000","last_peered":"2024-09-18T12:03:49.058635+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:49.058635+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4723922,"num_objects":1155,"num_object_clones":0,"num_object_copies":2310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18217,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18476","reported_seq":18704,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:49.096593+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:49.096593+0000","last_peered":"2024-09-18T12:03:49.096593+0000","last_clean":"2024-09-18T12:03:49.096593+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:49.096593+0000","last_undegraded":"2024-09-18T12:03:49.096593+0000","last_fullsized":"2024-09-18T12:03:49.096593+0000","mapping_epoch":29,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4612096,"num_objects":1126,"num_object_clones":0,"num_object_copies":2252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14793,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18157","reported_seq":25289,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:45.986688+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:45.986688+0000","last_peered":"2024-09-18T12:03:45.986688+0000","last_clean":"2024-09-18T12:03:45.986688+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:45.986688+0000","last_undegraded":"2024-09-18T12:03:45.986688+0000","last_fullsized":"2024-09-18T12:03:45.986688+0000","mapping_epoch":38,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5113042,"num_objects":1250,"num_object_clones":0,"num_object_copies":2500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18155,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18292","reported_seq":22176,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.118523+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:49.118523+0000","last_peered":"2024-09-18T12:03:49.118523+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:49.118523+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4728018,"num_objects":1156,"num_object_clones":0,"num_object_copies":2312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18290,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1903,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-477}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18299","reported_seq":18359,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:48.822593+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:48.822593+0000","last_peered":"2024-09-18T12:03:48.822593+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:48.822593+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4768978,"num_objects":1166,"num_object_clones":0,"num_object_copies":2332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18297,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2384}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18353","reported_seq":25765,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:48.868549+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:48.868549+0000","last_peered":"2024-09-18T12:03:48.868549+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:48.868549+0000","last_undegraded":"2024-09-18T12:03:48.868549+0000","last_fullsized":"2024-09-18T12:03:48.868549+0000","mapping_epoch":42,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4689920,"num_objects":1145,"num_object_clones":0,"num_object_copies":2290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14833,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1145}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18333","reported_seq":18419,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.122755+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:49.122755+0000","last_peered":"2024-09-18T12:03:49.122755+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:49.122755+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4895337,"num_objects":1196,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18332,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18320","reported_seq":25470,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:45.916051+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:45.916051+0000","last_peered":"2024-09-18T12:03:45.916051+0000","last_clean":"2024-09-18T12:03:45.916051+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:45.916051+0000","last_undegraded":"2024-09-18T12:03:45.916051+0000","last_fullsized":"2024-09-18T12:03:45.916051+0000","mapping_epoch":40,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4925243,"num_objects":1205,"num_object_clones":0,"num_object_copies":2410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18317,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18104","reported_seq":18189,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.080353+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:49.080353+0000","last_peered":"2024-09-18T12:03:49.080353+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:49.080353+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4482258,"num_objects":1096,"num_object_clones":0,"num_object_copies":2192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18102,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2497}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18398","reported_seq":18487,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:48.989113+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:48.989113+0000","last_peered":"2024-09-18T12:03:48.989113+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:48.989113+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4565412,"num_objects":1118,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18395,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":75247692,"num_objects":18389,"num_object_clones":0,"num_object_copies":36778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18389,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":276826,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12525,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327647232,"data_stored":327583424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160933,"ondisk_log_size":160933,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051703,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456920,"kb_used_data":150732,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93914920,"statfs":{"total":96636764160,"available":96168878080,"internally_reserved":0,"allocated":154349568,"data_stored":154100933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.34000000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.187}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.221},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.373}]}]},{"osd":1,"up_from":35,"seq":150323855465,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":366308,"kb_used_data":60120,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94005532,"statfs":{"total":96636764160,"available":96261664768,"internally_reserved":0,"allocated":61562880,"data_stored":61265396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.499}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.34999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409204,"kb_used_data":111404,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93962636,"statfs":{"total":96636764160,"available":96217739264,"internally_reserved":0,"allocated":114077696,"data_stored":113829511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,3,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.74199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113668096,"data_stored":113664617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60428288,"data_stored":60375510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153550848,"data_stored":153543297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:50.860 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:03:50.860 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'18394", 'reported_seq': 18469, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:49.077754+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:03:49.077754+0000', 'last_peered': '2024-09-18T12:03:49.077754+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:03:49.077754+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'8300", 'ondisk_log_start': "39'8300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4677632, 'num_objects': 1142, 'num_object_clones': 0, 'num_object_copies': 2284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18394, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2457}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:50.860 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:03:50.860 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'18451", 'reported_seq': 18641, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:48.876051+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:03:48.876051+0000', 'last_peered': '2024-09-18T12:03:48.876051+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:03:48.876051+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'8400", 'ondisk_log_start': "39'8400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4530200, 'num_objects': 1107, 'num_object_clones': 0, 'num_object_copies': 2214, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1107, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15309, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2429}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:50.860 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:03:50.860 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'18274", 'reported_seq': 18359, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:49.085459+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:03:49.085459+0000', 'last_peered': '2024-09-18T12:03:49.085459+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:03:49.085459+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'8200", 'ondisk_log_start': "39'8200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4707538, 'num_objects': 1151, 'num_object_clones': 0, 'num_object_copies': 2302, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18272, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2376}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:50.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:03:50.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'18219", 'reported_seq': 18310, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:49.058635+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:03:49.058635+0000', 'last_peered': '2024-09-18T12:03:49.058635+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:03:49.058635+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'8200", 'ondisk_log_start': "39'8200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4723922, 'num_objects': 1155, 'num_object_clones': 0, 'num_object_copies': 2310, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1155, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18217, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:50.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:03:50.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'18292", 'reported_seq': 22176, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:49.118523+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:03:49.118523+0000', 'last_peered': '2024-09-18T12:03:49.118523+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:03:49.118523+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'8200", 'ondisk_log_start': "39'8200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4728018, 'num_objects': 1156, 'num_object_clones': 0, 'num_object_copies': 2312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1633, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1156, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18290, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1903, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -477}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:50.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:03:50.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'18299", 'reported_seq': 18359, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:48.822593+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:03:48.822593+0000', 'last_peered': '2024-09-18T12:03:48.822593+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:03:48.822593+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'8200", 'ondisk_log_start': "39'8200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4768978, 'num_objects': 1166, 'num_object_clones': 0, 'num_object_copies': 2332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18297, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2384}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:50.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:03:50.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'18353", 'reported_seq': 25765, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:03:48.868549+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:03:48.868549+0000', 'last_peered': '2024-09-18T12:03:48.868549+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:03:48.868549+0000', 'last_undegraded': '2024-09-18T12:03:48.868549+0000', 'last_fullsized': '2024-09-18T12:03:48.868549+0000', 'mapping_epoch': 42, 'log_start': "39'8300", 'ondisk_log_start': "39'8300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4689920, 'num_objects': 1145, 'num_object_clones': 0, 'num_object_copies': 2290, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1145, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14833, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1145}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:50.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:03:50.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'18333", 'reported_seq': 18419, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:49.122755+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:03:49.122755+0000', 'last_peered': '2024-09-18T12:03:49.122755+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:03:49.122755+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'8300", 'ondisk_log_start': "39'8300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4895337, 'num_objects': 1196, 'num_object_clones': 0, 'num_object_copies': 2392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18332, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2339}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:50.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:03:50.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'18104", 'reported_seq': 18189, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:49.080353+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:03:49.080353+0000', 'last_peered': '2024-09-18T12:03:49.080353+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:03:49.080353+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'8100", 'ondisk_log_start': "39'8100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4482258, 'num_objects': 1096, 'num_object_clones': 0, 'num_object_copies': 2192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18102, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2497}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:50.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:03:50.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'18398", 'reported_seq': 18487, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:48.989113+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:03:48.989113+0000', 'last_peered': '2024-09-18T12:03:48.989113+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:03:48.989113+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'8300", 'ondisk_log_start': "39'8300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4565412, 'num_objects': 1118, 'num_object_clones': 0, 'num_object_copies': 2236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18395, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2405}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:50.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:03:51.633 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:51.633 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30068/36782 objects degraded (81.747%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":631,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":18391,"data_bytes":75247711,"bytes_used":1262010368,"bytes_avail":288648282112,"bytes_total":289910292480,"degraded_objects":30068,"degraded_total":36782,"degraded_ratio":0.81746506443369038,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":274,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":525},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:03:51.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:03:51.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:52.421 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:52.429 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:52.475 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-09-18T12:03:51.250108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74133599,"num_objects":18119,"num_object_clones":0,"num_object_copies":36238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277100,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12526,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161008,"ondisk_log_size":161008,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1234200,"kb_used_data":315708,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":281881320,"statfs":{"total":289910292480,"available":288646471680,"internally_reserved":0,"allocated":323284992,"data_stored":322490688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,6,6,8,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19042304,"num_objects":-4649,"num_object_clones":0,"num_object_copies":-9298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002684"},"pg_stats":[{"pgid":"2.f","version":"43'18394","reported_seq":18469,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.077754+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:49.077754+0000","last_peered":"2024-09-18T12:03:49.077754+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:49.077754+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4677632,"num_objects":1142,"num_object_clones":0,"num_object_copies":2284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18394,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2457}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18296","reported_seq":18458,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:48.895397+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:48.895397+0000","last_peered":"2024-09-18T12:03:48.895397+0000","last_clean":"2024-09-18T12:03:48.895397+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:48.895397+0000","last_undegraded":"2024-09-18T12:03:48.895397+0000","last_fullsized":"2024-09-18T12:03:48.895397+0000","mapping_epoch":29,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":1136,"num_object_clones":0,"num_object_copies":2272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15488,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18451","reported_seq":18641,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:48.876051+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:48.876051+0000","last_peered":"2024-09-18T12:03:48.876051+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:48.876051+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4530200,"num_objects":1107,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15309,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2429}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18139","reported_seq":18170,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:48.929508+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:48.929508+0000","last_peered":"2024-09-18T12:03:48.929508+0000","last_clean":"2024-09-18T12:03:48.929508+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:48.929508+0000","last_undegraded":"2024-09-18T12:03:48.929508+0000","last_fullsized":"2024-09-18T12:03:48.929508+0000","mapping_epoch":29,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4591616,"num_objects":1121,"num_object_clones":0,"num_object_copies":2242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18139,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18328","reported_seq":18523,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:49.086476+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:49.086476+0000","last_peered":"2024-09-18T12:03:49.086476+0000","last_clean":"2024-09-18T12:03:49.086476+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:49.086476+0000","last_undegraded":"2024-09-18T12:03:49.086476+0000","last_fullsized":"2024-09-18T12:03:49.086476+0000","mapping_epoch":29,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4583424,"num_objects":1119,"num_object_clones":0,"num_object_copies":2238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15493,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18274","reported_seq":18359,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.085459+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:49.085459+0000","last_peered":"2024-09-18T12:03:49.085459+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:49.085459+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4707538,"num_objects":1151,"num_object_clones":0,"num_object_copies":2302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18272,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18219","reported_seq":18310,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.058635+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:49.058635+0000","last_peered":"2024-09-18T12:03:49.058635+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:49.058635+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4723922,"num_objects":1155,"num_object_clones":0,"num_object_copies":2310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18217,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18476","reported_seq":18704,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:49.096593+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:49.096593+0000","last_peered":"2024-09-18T12:03:49.096593+0000","last_clean":"2024-09-18T12:03:49.096593+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:49.096593+0000","last_undegraded":"2024-09-18T12:03:49.096593+0000","last_fullsized":"2024-09-18T12:03:49.096593+0000","mapping_epoch":29,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4612096,"num_objects":1126,"num_object_clones":0,"num_object_copies":2252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14793,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18284","reported_seq":25416,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:50.949500+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:50.949500+0000","last_peered":"2024-09-18T12:03:50.949500+0000","last_clean":"2024-09-18T12:03:50.949500+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:50.949500+0000","last_undegraded":"2024-09-18T12:03:50.949500+0000","last_fullsized":"2024-09-18T12:03:50.949500+0000","mapping_epoch":38,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4592850,"num_objects":1123,"num_object_clones":0,"num_object_copies":2246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18282,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18292","reported_seq":22176,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.118523+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:49.118523+0000","last_peered":"2024-09-18T12:03:49.118523+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:49.118523+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4728018,"num_objects":1156,"num_object_clones":0,"num_object_copies":2312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18290,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1903,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-477}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18299","reported_seq":18359,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:48.822593+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:48.822593+0000","last_peered":"2024-09-18T12:03:48.822593+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:48.822593+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4768978,"num_objects":1166,"num_object_clones":0,"num_object_copies":2332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18297,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2384}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18353","reported_seq":25765,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:48.868549+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:48.868549+0000","last_peered":"2024-09-18T12:03:48.868549+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:48.868549+0000","last_undegraded":"2024-09-18T12:03:48.868549+0000","last_fullsized":"2024-09-18T12:03:48.868549+0000","mapping_epoch":42,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4689920,"num_objects":1145,"num_object_clones":0,"num_object_copies":2290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14833,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1145}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18333","reported_seq":18419,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.122755+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:49.122755+0000","last_peered":"2024-09-18T12:03:49.122755+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:49.122755+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4895337,"num_objects":1196,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18332,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18465","reported_seq":25615,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:51.014944+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:51.014944+0000","last_peered":"2024-09-18T12:03:51.014944+0000","last_clean":"2024-09-18T12:03:51.014944+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:51.014944+0000","last_undegraded":"2024-09-18T12:03:51.014944+0000","last_fullsized":"2024-09-18T12:03:51.014944+0000","mapping_epoch":40,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4331323,"num_objects":1060,"num_object_clones":0,"num_object_copies":2120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18462,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18104","reported_seq":18189,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:49.080353+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:49.080353+0000","last_peered":"2024-09-18T12:03:49.080353+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:49.080353+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8100","ondisk_log_start":"39'8100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4482258,"num_objects":1096,"num_object_clones":0,"num_object_copies":2192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18102,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2497}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18398","reported_seq":18487,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:48.989113+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:48.989113+0000","last_peered":"2024-09-18T12:03:48.989113+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:48.989113+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4565412,"num_objects":1118,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18395,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":74133580,"num_objects":18117,"num_object_clones":0,"num_object_copies":36234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277098,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12525,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":320831488,"data_stored":320767680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161005,"ondisk_log_size":161005,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051703,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456920,"kb_used_data":150732,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93914920,"statfs":{"total":96636764160,"available":96168878080,"internally_reserved":0,"allocated":154349568,"data_stored":154100933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.34000000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.187}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.221},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.373}]}]},{"osd":1,"up_from":35,"seq":150323855466,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368076,"kb_used_data":53572,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":94003764,"statfs":{"total":96636764160,"available":96259854336,"internally_reserved":0,"allocated":54857728,"data_stored":54560244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.40300000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409204,"kb_used_data":111404,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93962636,"statfs":{"total":96636764160,"available":96217739264,"internally_reserved":0,"allocated":114077696,"data_stored":113829511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,3,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.74199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":113668096,"data_stored":113664617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53612544,"data_stored":53559766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153550848,"data_stored":153543297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:55.477 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:56.222 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:56.252 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:56.274 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-18T12:03:55.251000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":66338911,"num_objects":16216,"num_object_clones":0,"num_object_copies":32432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16216,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":279003,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12617,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160811,"ondisk_log_size":160811,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1232896,"kb_used_data":306088,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281882624,"statfs":{"total":289910292480,"available":288647806976,"internally_reserved":0,"allocated":313434112,"data_stored":312639808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,11,3,7,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25812992,"num_objects":-6302,"num_object_clones":0,"num_object_copies":-12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6302,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002643"},"pg_stats":[{"pgid":"2.f","version":"43'18527","reported_seq":18602,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:53.998253+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:53.998253+0000","last_peered":"2024-09-18T12:03:53.998253+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:53.998253+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4132864,"num_objects":1009,"num_object_clones":0,"num_object_copies":2018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18527,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2590}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18437","reported_seq":18599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:53.919117+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:53.919117+0000","last_peered":"2024-09-18T12:03:53.919117+0000","last_clean":"2024-09-18T12:03:53.919117+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:53.919117+0000","last_undegraded":"2024-09-18T12:03:53.919117+0000","last_fullsized":"2024-09-18T12:03:53.919117+0000","mapping_epoch":29,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4075520,"num_objects":995,"num_object_clones":0,"num_object_copies":1990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15629,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18564","reported_seq":18754,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:53.867850+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:53.867850+0000","last_peered":"2024-09-18T12:03:53.867850+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:53.867850+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4067352,"num_objects":994,"num_object_clones":0,"num_object_copies":1988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15422,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2542}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18271","reported_seq":18304,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:53.750505+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:53.750505+0000","last_peered":"2024-09-18T12:03:53.750505+0000","last_clean":"2024-09-18T12:03:53.750505+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:53.750505+0000","last_undegraded":"2024-09-18T12:03:53.750505+0000","last_fullsized":"2024-09-18T12:03:53.750505+0000","mapping_epoch":29,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4050944,"num_objects":989,"num_object_clones":0,"num_object_copies":1978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18271,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18466","reported_seq":18661,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:54.130881+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:54.130881+0000","last_peered":"2024-09-18T12:03:54.130881+0000","last_clean":"2024-09-18T12:03:54.130881+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:54.130881+0000","last_undegraded":"2024-09-18T12:03:54.130881+0000","last_fullsized":"2024-09-18T12:03:54.130881+0000","mapping_epoch":29,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4018176,"num_objects":981,"num_object_clones":0,"num_object_copies":1962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15631,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18401","reported_seq":18486,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.000211+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:54.000211+0000","last_peered":"2024-09-18T12:03:54.000211+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:54.000211+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4187346,"num_objects":1024,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18399,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18373","reported_seq":18464,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.091604+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:54.091604+0000","last_peered":"2024-09-18T12:03:54.091604+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:54.091604+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4093138,"num_objects":1001,"num_object_clones":0,"num_object_copies":2002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18371,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2558}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18622","reported_seq":18851,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:54.011681+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:54.011681+0000","last_peered":"2024-09-18T12:03:54.011681+0000","last_clean":"2024-09-18T12:03:54.011681+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:54.011681+0000","last_undegraded":"2024-09-18T12:03:54.011681+0000","last_fullsized":"2024-09-18T12:03:54.011681+0000","mapping_epoch":29,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4014080,"num_objects":980,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14939,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18284","reported_seq":25416,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:50.949500+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:50.949500+0000","last_peered":"2024-09-18T12:03:50.949500+0000","last_clean":"2024-09-18T12:03:50.949500+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:50.949500+0000","last_undegraded":"2024-09-18T12:03:50.949500+0000","last_fullsized":"2024-09-18T12:03:50.949500+0000","mapping_epoch":38,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4592850,"num_objects":1123,"num_object_clones":0,"num_object_copies":2246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18282,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18440","reported_seq":22507,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.107031+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:54.107031+0000","last_peered":"2024-09-18T12:03:54.107031+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:54.107031+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4121810,"num_objects":1008,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18438,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-534}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18433","reported_seq":18493,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:53.958253+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:53.958253+0000","last_peered":"2024-09-18T12:03:53.958253+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:53.958253+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4220114,"num_objects":1032,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18431,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2518}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18491","reported_seq":25904,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:53.622109+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:53.622109+0000","last_peered":"2024-09-18T12:03:53.622109+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:53.622109+0000","last_undegraded":"2024-09-18T12:03:53.622109+0000","last_fullsized":"2024-09-18T12:03:53.622109+0000","mapping_epoch":42,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4124672,"num_objects":1007,"num_object_clones":0,"num_object_copies":2014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14971,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1007}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18464","reported_seq":18549,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.091602+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:54.091602+0000","last_peered":"2024-09-18T12:03:54.091602+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:54.091602+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4358761,"num_objects":1065,"num_object_clones":0,"num_object_copies":2130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18463,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2470}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18465","reported_seq":25615,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:51.014944+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:51.014944+0000","last_peered":"2024-09-18T12:03:51.014944+0000","last_clean":"2024-09-18T12:03:51.014944+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:51.014944+0000","last_undegraded":"2024-09-18T12:03:51.014944+0000","last_fullsized":"2024-09-18T12:03:51.014944+0000","mapping_epoch":40,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4331323,"num_objects":1060,"num_object_clones":0,"num_object_copies":2120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18462,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18238","reported_seq":18323,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.039255+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:54.039255+0000","last_peered":"2024-09-18T12:03:54.039255+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:54.039255+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3933394,"num_objects":962,"num_object_clones":0,"num_object_copies":1924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18236,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18532","reported_seq":18619,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.037112+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:54.037112+0000","last_peered":"2024-09-18T12:03:54.037112+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:54.037112+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4016548,"num_objects":984,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18529,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2539}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":66338892,"num_objects":16214,"num_object_clones":0,"num_object_copies":32428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16214,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":279001,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12616,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":309653504,"data_stored":309589696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160808,"ondisk_log_size":160808,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051704,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456848,"kb_used_data":146504,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93914992,"statfs":{"total":96636764160,"available":96168951808,"internally_reserved":0,"allocated":150020096,"data_stored":149771461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.252},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.34200000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.28399999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855466,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368076,"kb_used_data":53572,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":94003764,"statfs":{"total":96636764160,"available":96259854336,"internally_reserved":0,"allocated":54857728,"data_stored":54560244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.40300000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247931,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407972,"kb_used_data":106012,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93963868,"statfs":{"total":96636764160,"available":96219000832,"internally_reserved":0,"allocated":108556288,"data_stored":108308103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.44},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":8.6430000000000007}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106860544,"data_stored":106857065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53612544,"data_stored":53559766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149180416,"data_stored":149172865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:56.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:56.905 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:56.934 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:56.950 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-18T12:03:55.251000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":66338911,"num_objects":16216,"num_object_clones":0,"num_object_copies":32432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16216,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":279003,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12617,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160811,"ondisk_log_size":160811,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1232896,"kb_used_data":306088,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":281882624,"statfs":{"total":289910292480,"available":288647806976,"internally_reserved":0,"allocated":313434112,"data_stored":312639808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,11,3,7,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25812992,"num_objects":-6302,"num_object_clones":0,"num_object_copies":-12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6302,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002643"},"pg_stats":[{"pgid":"2.f","version":"43'18527","reported_seq":18602,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:53.998253+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:53.998253+0000","last_peered":"2024-09-18T12:03:53.998253+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:53.998253+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4132864,"num_objects":1009,"num_object_clones":0,"num_object_copies":2018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18527,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2590}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18437","reported_seq":18599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:53.919117+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:53.919117+0000","last_peered":"2024-09-18T12:03:53.919117+0000","last_clean":"2024-09-18T12:03:53.919117+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:53.919117+0000","last_undegraded":"2024-09-18T12:03:53.919117+0000","last_fullsized":"2024-09-18T12:03:53.919117+0000","mapping_epoch":29,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4075520,"num_objects":995,"num_object_clones":0,"num_object_copies":1990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15629,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18564","reported_seq":18754,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:53.867850+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:53.867850+0000","last_peered":"2024-09-18T12:03:53.867850+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:53.867850+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4067352,"num_objects":994,"num_object_clones":0,"num_object_copies":1988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15422,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2542}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18271","reported_seq":18304,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:53.750505+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:53.750505+0000","last_peered":"2024-09-18T12:03:53.750505+0000","last_clean":"2024-09-18T12:03:53.750505+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:53.750505+0000","last_undegraded":"2024-09-18T12:03:53.750505+0000","last_fullsized":"2024-09-18T12:03:53.750505+0000","mapping_epoch":29,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4050944,"num_objects":989,"num_object_clones":0,"num_object_copies":1978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18271,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18466","reported_seq":18661,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:54.130881+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:54.130881+0000","last_peered":"2024-09-18T12:03:54.130881+0000","last_clean":"2024-09-18T12:03:54.130881+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:54.130881+0000","last_undegraded":"2024-09-18T12:03:54.130881+0000","last_fullsized":"2024-09-18T12:03:54.130881+0000","mapping_epoch":29,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4018176,"num_objects":981,"num_object_clones":0,"num_object_copies":1962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15631,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18401","reported_seq":18486,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.000211+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:54.000211+0000","last_peered":"2024-09-18T12:03:54.000211+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:54.000211+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4187346,"num_objects":1024,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18399,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18373","reported_seq":18464,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.091604+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:54.091604+0000","last_peered":"2024-09-18T12:03:54.091604+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:54.091604+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4093138,"num_objects":1001,"num_object_clones":0,"num_object_copies":2002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18371,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2558}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18622","reported_seq":18851,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:54.011681+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:54.011681+0000","last_peered":"2024-09-18T12:03:54.011681+0000","last_clean":"2024-09-18T12:03:54.011681+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:54.011681+0000","last_undegraded":"2024-09-18T12:03:54.011681+0000","last_fullsized":"2024-09-18T12:03:54.011681+0000","mapping_epoch":29,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4014080,"num_objects":980,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14939,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18284","reported_seq":25416,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:50.949500+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:50.949500+0000","last_peered":"2024-09-18T12:03:50.949500+0000","last_clean":"2024-09-18T12:03:50.949500+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:50.949500+0000","last_undegraded":"2024-09-18T12:03:50.949500+0000","last_fullsized":"2024-09-18T12:03:50.949500+0000","mapping_epoch":38,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4592850,"num_objects":1123,"num_object_clones":0,"num_object_copies":2246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18282,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18440","reported_seq":22507,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.107031+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:54.107031+0000","last_peered":"2024-09-18T12:03:54.107031+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:54.107031+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4121810,"num_objects":1008,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18438,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-534}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18433","reported_seq":18493,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:53.958253+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:53.958253+0000","last_peered":"2024-09-18T12:03:53.958253+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:53.958253+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4220114,"num_objects":1032,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18431,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2518}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18491","reported_seq":25904,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:53.622109+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:53.622109+0000","last_peered":"2024-09-18T12:03:53.622109+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:53.622109+0000","last_undegraded":"2024-09-18T12:03:53.622109+0000","last_fullsized":"2024-09-18T12:03:53.622109+0000","mapping_epoch":42,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4124672,"num_objects":1007,"num_object_clones":0,"num_object_copies":2014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14971,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1007}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18464","reported_seq":18549,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.091602+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:54.091602+0000","last_peered":"2024-09-18T12:03:54.091602+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:54.091602+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4358761,"num_objects":1065,"num_object_clones":0,"num_object_copies":2130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18463,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2470}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18465","reported_seq":25615,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:51.014944+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:51.014944+0000","last_peered":"2024-09-18T12:03:51.014944+0000","last_clean":"2024-09-18T12:03:51.014944+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:51.014944+0000","last_undegraded":"2024-09-18T12:03:51.014944+0000","last_fullsized":"2024-09-18T12:03:51.014944+0000","mapping_epoch":40,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4331323,"num_objects":1060,"num_object_clones":0,"num_object_copies":2120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18462,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18238","reported_seq":18323,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.039255+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:54.039255+0000","last_peered":"2024-09-18T12:03:54.039255+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:54.039255+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3933394,"num_objects":962,"num_object_clones":0,"num_object_copies":1924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18236,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18532","reported_seq":18619,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.037112+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:54.037112+0000","last_peered":"2024-09-18T12:03:54.037112+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:54.037112+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4016548,"num_objects":984,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18529,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2539}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":66338892,"num_objects":16214,"num_object_clones":0,"num_object_copies":32428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16214,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":279001,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12616,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":309653504,"data_stored":309589696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160808,"ondisk_log_size":160808,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051704,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456848,"kb_used_data":146504,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93914992,"statfs":{"total":96636764160,"available":96168951808,"internally_reserved":0,"allocated":150020096,"data_stored":149771461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.252},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.34200000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.28399999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855466,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368076,"kb_used_data":53572,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":94003764,"statfs":{"total":96636764160,"available":96259854336,"internally_reserved":0,"allocated":54857728,"data_stored":54560244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.40300000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247931,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407972,"kb_used_data":106012,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93963868,"statfs":{"total":96636764160,"available":96219000832,"internally_reserved":0,"allocated":108556288,"data_stored":108308103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.44},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":8.6430000000000007}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106860544,"data_stored":106857065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53612544,"data_stored":53559766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149180416,"data_stored":149172865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:03:56.951 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:03:56.951 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'18527", 'reported_seq': 18602, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:53.998253+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:03:53.998253+0000', 'last_peered': '2024-09-18T12:03:53.998253+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:03:53.998253+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'8500", 'ondisk_log_start': "39'8500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4132864, 'num_objects': 1009, 'num_object_clones': 0, 'num_object_copies': 2018, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1009, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18527, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2590}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:56.951 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:03:56.952 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'18564", 'reported_seq': 18754, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:53.867850+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:03:53.867850+0000', 'last_peered': '2024-09-18T12:03:53.867850+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:03:53.867850+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'8500", 'ondisk_log_start': "39'8500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4067352, 'num_objects': 994, 'num_object_clones': 0, 'num_object_copies': 1988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 994, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15422, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2542}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:56.952 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:03:56.952 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'18401", 'reported_seq': 18486, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:54.000211+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:03:54.000211+0000', 'last_peered': '2024-09-18T12:03:54.000211+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:03:54.000211+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'8400", 'ondisk_log_start': "39'8400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4187346, 'num_objects': 1024, 'num_object_clones': 0, 'num_object_copies': 2048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18399, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:56.952 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:03:56.952 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'18373", 'reported_seq': 18464, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:54.091604+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:03:54.091604+0000', 'last_peered': '2024-09-18T12:03:54.091604+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:03:54.091604+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'8300", 'ondisk_log_start': "39'8300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4093138, 'num_objects': 1001, 'num_object_clones': 0, 'num_object_copies': 2002, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18371, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2558}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:56.952 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:03:56.952 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'18440", 'reported_seq': 22507, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:54.107031+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:03:54.107031+0000', 'last_peered': '2024-09-18T12:03:54.107031+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:03:54.107031+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'8400", 'ondisk_log_start': "39'8400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4121810, 'num_objects': 1008, 'num_object_clones': 0, 'num_object_copies': 2016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1542, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18438, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1994, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -534}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:56.952 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:03:56.952 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'18433", 'reported_seq': 18493, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:53.958253+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:03:53.958253+0000', 'last_peered': '2024-09-18T12:03:53.958253+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:03:53.958253+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'8400", 'ondisk_log_start': "39'8400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4220114, 'num_objects': 1032, 'num_object_clones': 0, 'num_object_copies': 2064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18431, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2518}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:56.953 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:03:56.953 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'18491", 'reported_seq': 25904, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:03:53.622109+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:03:53.622109+0000', 'last_peered': '2024-09-18T12:03:53.622109+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:03:53.622109+0000', 'last_undegraded': '2024-09-18T12:03:53.622109+0000', 'last_fullsized': '2024-09-18T12:03:53.622109+0000', 'mapping_epoch': 42, 'log_start': "39'8400", 'ondisk_log_start': "39'8400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4124672, 'num_objects': 1007, 'num_object_clones': 0, 'num_object_copies': 2014, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1007, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14971, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1007}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:03:56.953 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:03:56.953 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'18464", 'reported_seq': 18549, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:54.091602+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:03:54.091602+0000', 'last_peered': '2024-09-18T12:03:54.091602+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:03:54.091602+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'8400", 'ondisk_log_start': "39'8400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4358761, 'num_objects': 1065, 'num_object_clones': 0, 'num_object_copies': 2130, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18463, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2470}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:56.953 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:03:56.953 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'18238", 'reported_seq': 18323, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:54.039255+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:03:54.039255+0000', 'last_peered': '2024-09-18T12:03:54.039255+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:03:54.039255+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'8200", 'ondisk_log_start': "39'8200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3933394, 'num_objects': 962, 'num_object_clones': 0, 'num_object_copies': 1924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 962, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18236, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2631}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:56.953 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:03:56.953 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'18532", 'reported_seq': 18619, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:54.037112+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:03:54.037112+0000', 'last_peered': '2024-09-18T12:03:54.037112+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:03:54.037112+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'8500", 'ondisk_log_start': "39'8500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4016548, 'num_objects': 984, 'num_object_clones': 0, 'num_object_copies': 1968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18529, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2539}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:03:56.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:03:57.722 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:57.722 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29977/32432 objects degraded (92.430%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":637,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":16216,"data_bytes":66338911,"bytes_used":1262485504,"bytes_avail":288647806976,"bytes_total":289910292480,"degraded_objects":29977,"degraded_total":32432,"degraded_ratio":0.92430315737543167,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":276,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":525},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:03:57.790 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:03:57.790 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:03:58.354 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:03:58.357 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:03:58.373 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-09-18T12:03:57.251505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":65237087,"num_objects":15947,"num_object_clones":0,"num_object_copies":31894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":279272,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12617,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160680,"ondisk_log_size":160680,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1213696,"kb_used_data":299360,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281901824,"statfs":{"total":289910292480,"available":288667467776,"internally_reserved":0,"allocated":306544640,"data_stored":305750336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,11,3,7,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18984960,"num_objects":-4635,"num_object_clones":0,"num_object_copies":-9270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4635,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002743"},"pg_stats":[{"pgid":"2.f","version":"43'18527","reported_seq":18602,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:53.998253+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:53.998253+0000","last_peered":"2024-09-18T12:03:53.998253+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:53.998253+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4132864,"num_objects":1009,"num_object_clones":0,"num_object_copies":2018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18527,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2590}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18437","reported_seq":18599,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:53.919117+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:53.919117+0000","last_peered":"2024-09-18T12:03:53.919117+0000","last_clean":"2024-09-18T12:03:53.919117+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:53.919117+0000","last_undegraded":"2024-09-18T12:03:53.919117+0000","last_fullsized":"2024-09-18T12:03:53.919117+0000","mapping_epoch":29,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4075520,"num_objects":995,"num_object_clones":0,"num_object_copies":1990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15629,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18564","reported_seq":18754,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:53.867850+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:53.867850+0000","last_peered":"2024-09-18T12:03:53.867850+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:53.867850+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4067352,"num_objects":994,"num_object_clones":0,"num_object_copies":1988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15422,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2542}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18271","reported_seq":18304,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:53.750505+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:53.750505+0000","last_peered":"2024-09-18T12:03:53.750505+0000","last_clean":"2024-09-18T12:03:53.750505+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:53.750505+0000","last_undegraded":"2024-09-18T12:03:53.750505+0000","last_fullsized":"2024-09-18T12:03:53.750505+0000","mapping_epoch":29,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4050944,"num_objects":989,"num_object_clones":0,"num_object_copies":1978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18271,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18466","reported_seq":18661,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:54.130881+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:54.130881+0000","last_peered":"2024-09-18T12:03:54.130881+0000","last_clean":"2024-09-18T12:03:54.130881+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:54.130881+0000","last_undegraded":"2024-09-18T12:03:54.130881+0000","last_fullsized":"2024-09-18T12:03:54.130881+0000","mapping_epoch":29,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4018176,"num_objects":981,"num_object_clones":0,"num_object_copies":1962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15631,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18401","reported_seq":18486,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.000211+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:54.000211+0000","last_peered":"2024-09-18T12:03:54.000211+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:54.000211+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4187346,"num_objects":1024,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18399,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18373","reported_seq":18464,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.091604+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:54.091604+0000","last_peered":"2024-09-18T12:03:54.091604+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:54.091604+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4093138,"num_objects":1001,"num_object_clones":0,"num_object_copies":2002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18371,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2558}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18622","reported_seq":18851,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:54.011681+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:54.011681+0000","last_peered":"2024-09-18T12:03:54.011681+0000","last_clean":"2024-09-18T12:03:54.011681+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:54.011681+0000","last_undegraded":"2024-09-18T12:03:54.011681+0000","last_fullsized":"2024-09-18T12:03:54.011681+0000","mapping_epoch":29,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4014080,"num_objects":980,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14939,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18413","reported_seq":25545,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:56.112384+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:03:56.112384+0000","last_peered":"2024-09-18T12:03:56.112384+0000","last_clean":"2024-09-18T12:03:56.112384+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:03:56.112384+0000","last_undegraded":"2024-09-18T12:03:56.112384+0000","last_fullsized":"2024-09-18T12:03:56.112384+0000","mapping_epoch":38,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4064466,"num_objects":994,"num_object_clones":0,"num_object_copies":1988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18411,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18440","reported_seq":22507,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.107031+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:54.107031+0000","last_peered":"2024-09-18T12:03:54.107031+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:54.107031+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4121810,"num_objects":1008,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18438,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-534}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18433","reported_seq":18493,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:53.958253+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:53.958253+0000","last_peered":"2024-09-18T12:03:53.958253+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:53.958253+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4220114,"num_objects":1032,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18431,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2518}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18491","reported_seq":25904,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:53.622109+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:53.622109+0000","last_peered":"2024-09-18T12:03:53.622109+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:53.622109+0000","last_undegraded":"2024-09-18T12:03:53.622109+0000","last_fullsized":"2024-09-18T12:03:53.622109+0000","mapping_epoch":42,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4124672,"num_objects":1007,"num_object_clones":0,"num_object_copies":2014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14971,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":1007}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18464","reported_seq":18549,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.091602+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:54.091602+0000","last_peered":"2024-09-18T12:03:54.091602+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:54.091602+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8400","ondisk_log_start":"39'8400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4358761,"num_objects":1065,"num_object_clones":0,"num_object_copies":2130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18463,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2470}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18605","reported_seq":25755,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:56.042169+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:03:56.042169+0000","last_peered":"2024-09-18T12:03:56.042169+0000","last_clean":"2024-09-18T12:03:56.042169+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:03:56.042169+0000","last_undegraded":"2024-09-18T12:03:56.042169+0000","last_fullsized":"2024-09-18T12:03:56.042169+0000","mapping_epoch":40,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3757883,"num_objects":920,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18602,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18238","reported_seq":18323,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.039255+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:54.039255+0000","last_peered":"2024-09-18T12:03:54.039255+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:54.039255+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8200","ondisk_log_start":"39'8200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3933394,"num_objects":962,"num_object_clones":0,"num_object_copies":1924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18236,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18532","reported_seq":18619,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:54.037112+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:54.037112+0000","last_peered":"2024-09-18T12:03:54.037112+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:54.037112+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4016548,"num_objects":984,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18529,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2539}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":65237068,"num_objects":15945,"num_object_clones":0,"num_object_copies":31890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15945,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":279270,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12616,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":302796800,"data_stored":302732992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160677,"ondisk_log_size":160677,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051704,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456848,"kb_used_data":146504,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93914992,"statfs":{"total":96636764160,"available":96168951808,"internally_reserved":0,"allocated":150020096,"data_stored":149771461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.252},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.34200000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.28399999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855467,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348876,"kb_used_data":46844,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94022964,"statfs":{"total":96636764160,"available":96279515136,"internally_reserved":0,"allocated":47968256,"data_stored":47670772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.40300000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247931,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407972,"kb_used_data":106012,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93963868,"statfs":{"total":96636764160,"available":96219000832,"internally_reserved":0,"allocated":108556288,"data_stored":108308103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.44},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":8.6430000000000007}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106860544,"data_stored":106857065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46755840,"data_stored":46703062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149180416,"data_stored":149172865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:01.376 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:02.131 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:02.169 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:02.202 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-09-18T12:04:01.252450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":56467551,"num_objects":13806,"num_object_clones":0,"num_object_copies":27612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":281413,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12710,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1213408,"kb_used_data":282436,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":281902112,"statfs":{"total":289910292480,"available":288667762688,"internally_reserved":0,"allocated":289214464,"data_stored":288420160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,5,7,15,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18780160,"num_objects":-4585,"num_object_clones":0,"num_object_copies":-9170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4585,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002788"},"pg_stats":[{"pgid":"2.f","version":"43'18685","reported_seq":18760,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.060105+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:59.060105+0000","last_peered":"2024-09-18T12:03:59.060105+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:59.060105+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3485696,"num_objects":851,"num_object_clones":0,"num_object_copies":1702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18685,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18566","reported_seq":18728,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:58.869557+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:58.869557+0000","last_peered":"2024-09-18T12:03:58.869557+0000","last_clean":"2024-09-18T12:03:58.869557+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:58.869557+0000","last_undegraded":"2024-09-18T12:03:58.869557+0000","last_fullsized":"2024-09-18T12:03:58.869557+0000","mapping_epoch":29,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3547136,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15758,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18691","reported_seq":18881,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:58.954990+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:58.954990+0000","last_peered":"2024-09-18T12:03:58.954990+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:58.954990+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3547160,"num_objects":867,"num_object_clones":0,"num_object_copies":1734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15549,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2669}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18394","reported_seq":18427,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:58.900319+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:58.900319+0000","last_peered":"2024-09-18T12:03:58.900319+0000","last_clean":"2024-09-18T12:03:58.900319+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:58.900319+0000","last_undegraded":"2024-09-18T12:03:58.900319+0000","last_fullsized":"2024-09-18T12:03:58.900319+0000","mapping_epoch":29,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3547136,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18394,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18587","reported_seq":18782,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:59.036755+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:59.036755+0000","last_peered":"2024-09-18T12:03:59.036755+0000","last_clean":"2024-09-18T12:03:59.036755+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:59.036755+0000","last_undegraded":"2024-09-18T12:03:59.036755+0000","last_fullsized":"2024-09-18T12:03:59.036755+0000","mapping_epoch":29,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3522560,"num_objects":860,"num_object_clones":0,"num_object_copies":1720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15752,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18540","reported_seq":18625,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.069543+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:59.069543+0000","last_peered":"2024-09-18T12:03:59.069543+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:59.069543+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3618002,"num_objects":885,"num_object_clones":0,"num_object_copies":1770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18538,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18508","reported_seq":18597,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.138829+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:59.138829+0000","last_peered":"2024-09-18T12:03:59.138829+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:59.138829+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3540178,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18506,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18772","reported_seq":19001,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:59.112713+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:59.112713+0000","last_peered":"2024-09-18T12:03:59.112713+0000","last_clean":"2024-09-18T12:03:59.112713+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:59.112713+0000","last_undegraded":"2024-09-18T12:03:59.112713+0000","last_fullsized":"2024-09-18T12:03:59.112713+0000","mapping_epoch":29,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3399680,"num_objects":830,"num_object_clones":0,"num_object_copies":1660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15089,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18547","reported_seq":25679,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:01.153765+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:01.153765+0000","last_peered":"2024-09-18T12:04:01.153765+0000","last_clean":"2024-09-18T12:04:01.153765+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:01.153765+0000","last_undegraded":"2024-09-18T12:04:01.153765+0000","last_fullsized":"2024-09-18T12:04:01.153765+0000","mapping_epoch":38,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3515602,"num_objects":860,"num_object_clones":0,"num_object_copies":1720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18545,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18575","reported_seq":22827,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.145747+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:59.145747+0000","last_peered":"2024-09-18T12:03:59.145747+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:59.145747+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3568850,"num_objects":873,"num_object_clones":0,"num_object_copies":1746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18573,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-576}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18563","reported_seq":18620,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:58.910289+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:58.910289+0000","last_peered":"2024-09-18T12:03:58.910289+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:58.910289+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3687634,"num_objects":902,"num_object_clones":0,"num_object_copies":1804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18561,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18638","reported_seq":26051,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:58.861235+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:58.861235+0000","last_peered":"2024-09-18T12:03:58.861235+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:58.861235+0000","last_undegraded":"2024-09-18T12:03:58.861235+0000","last_fullsized":"2024-09-18T12:03:58.861235+0000","mapping_epoch":42,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3522560,"num_objects":860,"num_object_clones":0,"num_object_copies":1720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15118,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":860}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18610","reported_seq":18694,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.061133+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:59.061133+0000","last_peered":"2024-09-18T12:03:59.061133+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:59.061133+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3760745,"num_objects":919,"num_object_clones":0,"num_object_copies":1838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18609,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18722","reported_seq":25871,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:01.147074+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:01.147074+0000","last_peered":"2024-09-18T12:04:01.147074+0000","last_clean":"2024-09-18T12:04:01.147074+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:01.147074+0000","last_undegraded":"2024-09-18T12:04:01.147074+0000","last_fullsized":"2024-09-18T12:04:01.147074+0000","mapping_epoch":40,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3278651,"num_objects":803,"num_object_clones":0,"num_object_copies":1606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18719,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18356","reported_seq":18440,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.044520+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:59.044520+0000","last_peered":"2024-09-18T12:03:59.044520+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:59.044520+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3450066,"num_objects":844,"num_object_clones":0,"num_object_copies":1688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18354,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18664","reported_seq":18753,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.086934+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:59.086934+0000","last_peered":"2024-09-18T12:03:59.086934+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:59.086934+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3475876,"num_objects":852,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18661,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":56467532,"num_objects":13804,"num_object_clones":0,"num_object_copies":27608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":281411,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12709,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":285179904,"data_stored":285116096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051705,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456900,"kb_used_data":142396,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93914940,"statfs":{"total":96636764160,"available":96168898560,"internally_reserved":0,"allocated":145813504,"data_stored":145564869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.30299999999999999},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.27700000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.26400000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855468,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350872,"kb_used_data":40528,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94020968,"statfs":{"total":96636764160,"available":96277471232,"internally_reserved":0,"allocated":41500672,"data_stored":41203188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":2.4700000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":2.3090000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405636,"kb_used_data":99512,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93966204,"statfs":{"total":96636764160,"available":96221392896,"internally_reserved":0,"allocated":101900288,"data_stored":101652103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,3,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.2440000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100278272,"data_stored":100274793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40034304,"data_stored":39981526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":144867328,"data_stored":144859777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:02.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:03.002 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:03.017 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:03.045 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-09-18T12:04:01.252450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":56467551,"num_objects":13806,"num_object_clones":0,"num_object_copies":27612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":281413,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12710,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1213408,"kb_used_data":282436,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":281902112,"statfs":{"total":289910292480,"available":288667762688,"internally_reserved":0,"allocated":289214464,"data_stored":288420160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,5,7,15,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18780160,"num_objects":-4585,"num_object_clones":0,"num_object_copies":-9170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4585,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002788"},"pg_stats":[{"pgid":"2.f","version":"43'18685","reported_seq":18760,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.060105+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:59.060105+0000","last_peered":"2024-09-18T12:03:59.060105+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:59.060105+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3485696,"num_objects":851,"num_object_clones":0,"num_object_copies":1702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18685,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18566","reported_seq":18728,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:58.869557+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:58.869557+0000","last_peered":"2024-09-18T12:03:58.869557+0000","last_clean":"2024-09-18T12:03:58.869557+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:58.869557+0000","last_undegraded":"2024-09-18T12:03:58.869557+0000","last_fullsized":"2024-09-18T12:03:58.869557+0000","mapping_epoch":29,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3547136,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15758,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18691","reported_seq":18881,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:58.954990+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:58.954990+0000","last_peered":"2024-09-18T12:03:58.954990+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:58.954990+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3547160,"num_objects":867,"num_object_clones":0,"num_object_copies":1734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15549,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2669}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18394","reported_seq":18427,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:58.900319+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:58.900319+0000","last_peered":"2024-09-18T12:03:58.900319+0000","last_clean":"2024-09-18T12:03:58.900319+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:58.900319+0000","last_undegraded":"2024-09-18T12:03:58.900319+0000","last_fullsized":"2024-09-18T12:03:58.900319+0000","mapping_epoch":29,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3547136,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18394,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18587","reported_seq":18782,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:59.036755+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:59.036755+0000","last_peered":"2024-09-18T12:03:59.036755+0000","last_clean":"2024-09-18T12:03:59.036755+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:59.036755+0000","last_undegraded":"2024-09-18T12:03:59.036755+0000","last_fullsized":"2024-09-18T12:03:59.036755+0000","mapping_epoch":29,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3522560,"num_objects":860,"num_object_clones":0,"num_object_copies":1720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15752,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18540","reported_seq":18625,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.069543+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:59.069543+0000","last_peered":"2024-09-18T12:03:59.069543+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:59.069543+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3618002,"num_objects":885,"num_object_clones":0,"num_object_copies":1770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18538,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18508","reported_seq":18597,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.138829+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:59.138829+0000","last_peered":"2024-09-18T12:03:59.138829+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:59.138829+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3540178,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18506,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18772","reported_seq":19001,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:59.112713+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:59.112713+0000","last_peered":"2024-09-18T12:03:59.112713+0000","last_clean":"2024-09-18T12:03:59.112713+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:59.112713+0000","last_undegraded":"2024-09-18T12:03:59.112713+0000","last_fullsized":"2024-09-18T12:03:59.112713+0000","mapping_epoch":29,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3399680,"num_objects":830,"num_object_clones":0,"num_object_copies":1660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15089,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18547","reported_seq":25679,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:01.153765+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:01.153765+0000","last_peered":"2024-09-18T12:04:01.153765+0000","last_clean":"2024-09-18T12:04:01.153765+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:01.153765+0000","last_undegraded":"2024-09-18T12:04:01.153765+0000","last_fullsized":"2024-09-18T12:04:01.153765+0000","mapping_epoch":38,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3515602,"num_objects":860,"num_object_clones":0,"num_object_copies":1720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18545,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18575","reported_seq":22827,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.145747+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:59.145747+0000","last_peered":"2024-09-18T12:03:59.145747+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:59.145747+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3568850,"num_objects":873,"num_object_clones":0,"num_object_copies":1746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18573,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-576}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18563","reported_seq":18620,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:58.910289+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:58.910289+0000","last_peered":"2024-09-18T12:03:58.910289+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:58.910289+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3687634,"num_objects":902,"num_object_clones":0,"num_object_copies":1804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18561,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18638","reported_seq":26051,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:58.861235+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:58.861235+0000","last_peered":"2024-09-18T12:03:58.861235+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:58.861235+0000","last_undegraded":"2024-09-18T12:03:58.861235+0000","last_fullsized":"2024-09-18T12:03:58.861235+0000","mapping_epoch":42,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3522560,"num_objects":860,"num_object_clones":0,"num_object_copies":1720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15118,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":860}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18610","reported_seq":18694,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.061133+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:59.061133+0000","last_peered":"2024-09-18T12:03:59.061133+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:59.061133+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3760745,"num_objects":919,"num_object_clones":0,"num_object_copies":1838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18609,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18722","reported_seq":25871,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:01.147074+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:01.147074+0000","last_peered":"2024-09-18T12:04:01.147074+0000","last_clean":"2024-09-18T12:04:01.147074+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:01.147074+0000","last_undegraded":"2024-09-18T12:04:01.147074+0000","last_fullsized":"2024-09-18T12:04:01.147074+0000","mapping_epoch":40,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3278651,"num_objects":803,"num_object_clones":0,"num_object_copies":1606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18719,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18356","reported_seq":18440,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.044520+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:59.044520+0000","last_peered":"2024-09-18T12:03:59.044520+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:59.044520+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3450066,"num_objects":844,"num_object_clones":0,"num_object_copies":1688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18354,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18664","reported_seq":18753,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.086934+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:59.086934+0000","last_peered":"2024-09-18T12:03:59.086934+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:59.086934+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3475876,"num_objects":852,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18661,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":56467532,"num_objects":13804,"num_object_clones":0,"num_object_copies":27608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":281411,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12709,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":285179904,"data_stored":285116096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051705,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456900,"kb_used_data":142396,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93914940,"statfs":{"total":96636764160,"available":96168898560,"internally_reserved":0,"allocated":145813504,"data_stored":145564869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.30299999999999999},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.27700000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.26400000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855468,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350872,"kb_used_data":40528,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94020968,"statfs":{"total":96636764160,"available":96277471232,"internally_reserved":0,"allocated":41500672,"data_stored":41203188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":2.4700000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":2.3090000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405636,"kb_used_data":99512,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93966204,"statfs":{"total":96636764160,"available":96221392896,"internally_reserved":0,"allocated":101900288,"data_stored":101652103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,3,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.2440000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100278272,"data_stored":100274793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40034304,"data_stored":39981526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":144867328,"data_stored":144859777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:03.046 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:04:03.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'18685", 'reported_seq': 18760, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:59.060105+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:03:59.060105+0000', 'last_peered': '2024-09-18T12:03:59.060105+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:03:59.060105+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'8600", 'ondisk_log_start': "39'8600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3485696, 'num_objects': 851, 'num_object_clones': 0, 'num_object_copies': 1702, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18685, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2748}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:03.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:04:03.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'18691", 'reported_seq': 18881, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:58.954990+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:03:58.954990+0000', 'last_peered': '2024-09-18T12:03:58.954990+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:03:58.954990+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'8600", 'ondisk_log_start': "39'8600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3547160, 'num_objects': 867, 'num_object_clones': 0, 'num_object_copies': 1734, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 867, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15549, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2669}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:03.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:04:03.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'18540", 'reported_seq': 18625, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:59.069543+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:03:59.069543+0000', 'last_peered': '2024-09-18T12:03:59.069543+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:03:59.069543+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'8500", 'ondisk_log_start': "39'8500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3618002, 'num_objects': 885, 'num_object_clones': 0, 'num_object_copies': 1770, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18538, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2642}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:03.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:04:03.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'18508", 'reported_seq': 18597, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:59.138829+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:03:59.138829+0000', 'last_peered': '2024-09-18T12:03:59.138829+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:03:59.138829+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'8500", 'ondisk_log_start': "39'8500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3540178, 'num_objects': 866, 'num_object_clones': 0, 'num_object_copies': 1732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 866, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18506, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2693}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:03.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:04:03.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'18575", 'reported_seq': 22827, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:59.145747+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:03:59.145747+0000', 'last_peered': '2024-09-18T12:03:59.145747+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:03:59.145747+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'8500", 'ondisk_log_start': "39'8500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3568850, 'num_objects': 873, 'num_object_clones': 0, 'num_object_copies': 1746, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1449, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18573, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2087, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -576}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:03.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:04:03.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'18563", 'reported_seq': 18620, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:58.910289+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:03:58.910289+0000', 'last_peered': '2024-09-18T12:03:58.910289+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:03:58.910289+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'8500", 'ondisk_log_start': "39'8500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3687634, 'num_objects': 902, 'num_object_clones': 0, 'num_object_copies': 1804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 902, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18561, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2648}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:03.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:04:03.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'18638", 'reported_seq': 26051, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:03:58.861235+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:03:58.861235+0000', 'last_peered': '2024-09-18T12:03:58.861235+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:03:58.861235+0000', 'last_undegraded': '2024-09-18T12:03:58.861235+0000', 'last_fullsized': '2024-09-18T12:03:58.861235+0000', 'mapping_epoch': 42, 'log_start': "39'8600", 'ondisk_log_start': "39'8600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3522560, 'num_objects': 860, 'num_object_clones': 0, 'num_object_copies': 1720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15118, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 860}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:03.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:04:03.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'18610", 'reported_seq': 18694, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:59.061133+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:03:59.061133+0000', 'last_peered': '2024-09-18T12:03:59.061133+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:03:59.061133+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'8600", 'ondisk_log_start': "39'8600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3760745, 'num_objects': 919, 'num_object_clones': 0, 'num_object_copies': 1838, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 919, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18609, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2616}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:03.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:04:03.049 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'18356", 'reported_seq': 18440, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:59.044520+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:03:59.044520+0000', 'last_peered': '2024-09-18T12:03:59.044520+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:03:59.044520+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'8300", 'ondisk_log_start': "39'8300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3450066, 'num_objects': 844, 'num_object_clones': 0, 'num_object_copies': 1688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18354, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2749}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:03.049 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:04:03.049 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'18664", 'reported_seq': 18753, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:03:59.086934+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:03:59.086934+0000', 'last_peered': '2024-09-18T12:03:59.086934+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:03:59.086934+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'8600", 'ondisk_log_start': "39'8600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3475876, 'num_objects': 852, 'num_object_clones': 0, 'num_object_copies': 1704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18661, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2671}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:03.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:04:03.686 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:03.686 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29884/27612 objects degraded (108.228%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":643,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":13806,"data_bytes":56467551,"bytes_used":1242529792,"bytes_avail":288667762688,"bytes_total":289910292480,"degraded_objects":29884,"degraded_total":27612,"degraded_ratio":1.0822830653339128,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":381},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:04:03.725 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:04:03.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:04.306 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:04.310 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:04.326 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-18T12:04:03.252902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":56467551,"num_objects":13806,"num_object_clones":0,"num_object_copies":27612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":281413,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12710,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1213408,"kb_used_data":282436,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":281902112,"statfs":{"total":289910292480,"available":288667762688,"internally_reserved":0,"allocated":289214464,"data_stored":288420160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,5,7,15,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17666048,"num_objects":-4313,"num_object_clones":0,"num_object_copies":-8626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4313,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002793"},"pg_stats":[{"pgid":"2.f","version":"43'18685","reported_seq":18760,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.060105+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:03:59.060105+0000","last_peered":"2024-09-18T12:03:59.060105+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:03:59.060105+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3485696,"num_objects":851,"num_object_clones":0,"num_object_copies":1702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18685,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18566","reported_seq":18728,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:58.869557+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:03:58.869557+0000","last_peered":"2024-09-18T12:03:58.869557+0000","last_clean":"2024-09-18T12:03:58.869557+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:03:58.869557+0000","last_undegraded":"2024-09-18T12:03:58.869557+0000","last_fullsized":"2024-09-18T12:03:58.869557+0000","mapping_epoch":29,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3547136,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15758,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18691","reported_seq":18881,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:58.954990+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:03:58.954990+0000","last_peered":"2024-09-18T12:03:58.954990+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:03:58.954990+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3547160,"num_objects":867,"num_object_clones":0,"num_object_copies":1734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15549,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2669}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18394","reported_seq":18427,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:58.900319+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:03:58.900319+0000","last_peered":"2024-09-18T12:03:58.900319+0000","last_clean":"2024-09-18T12:03:58.900319+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:03:58.900319+0000","last_undegraded":"2024-09-18T12:03:58.900319+0000","last_fullsized":"2024-09-18T12:03:58.900319+0000","mapping_epoch":29,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3547136,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18394,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18587","reported_seq":18782,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:59.036755+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:03:59.036755+0000","last_peered":"2024-09-18T12:03:59.036755+0000","last_clean":"2024-09-18T12:03:59.036755+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:03:59.036755+0000","last_undegraded":"2024-09-18T12:03:59.036755+0000","last_fullsized":"2024-09-18T12:03:59.036755+0000","mapping_epoch":29,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3522560,"num_objects":860,"num_object_clones":0,"num_object_copies":1720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15752,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18540","reported_seq":18625,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.069543+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:03:59.069543+0000","last_peered":"2024-09-18T12:03:59.069543+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:03:59.069543+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3618002,"num_objects":885,"num_object_clones":0,"num_object_copies":1770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18538,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18508","reported_seq":18597,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.138829+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:03:59.138829+0000","last_peered":"2024-09-18T12:03:59.138829+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:03:59.138829+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3540178,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18506,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18772","reported_seq":19001,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:03:59.112713+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:03:59.112713+0000","last_peered":"2024-09-18T12:03:59.112713+0000","last_clean":"2024-09-18T12:03:59.112713+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:03:59.112713+0000","last_undegraded":"2024-09-18T12:03:59.112713+0000","last_fullsized":"2024-09-18T12:03:59.112713+0000","mapping_epoch":29,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3399680,"num_objects":830,"num_object_clones":0,"num_object_copies":1660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15089,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18547","reported_seq":25679,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:01.153765+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:01.153765+0000","last_peered":"2024-09-18T12:04:01.153765+0000","last_clean":"2024-09-18T12:04:01.153765+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:01.153765+0000","last_undegraded":"2024-09-18T12:04:01.153765+0000","last_fullsized":"2024-09-18T12:04:01.153765+0000","mapping_epoch":38,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3515602,"num_objects":860,"num_object_clones":0,"num_object_copies":1720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18545,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18575","reported_seq":22827,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.145747+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:03:59.145747+0000","last_peered":"2024-09-18T12:03:59.145747+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:03:59.145747+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3568850,"num_objects":873,"num_object_clones":0,"num_object_copies":1746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18573,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-576}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18563","reported_seq":18620,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:58.910289+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:03:58.910289+0000","last_peered":"2024-09-18T12:03:58.910289+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:03:58.910289+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3687634,"num_objects":902,"num_object_clones":0,"num_object_copies":1804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18561,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18638","reported_seq":26051,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:03:58.861235+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:03:58.861235+0000","last_peered":"2024-09-18T12:03:58.861235+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:03:58.861235+0000","last_undegraded":"2024-09-18T12:03:58.861235+0000","last_fullsized":"2024-09-18T12:03:58.861235+0000","mapping_epoch":42,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3522560,"num_objects":860,"num_object_clones":0,"num_object_copies":1720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15118,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":860}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18610","reported_seq":18694,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.061133+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:03:59.061133+0000","last_peered":"2024-09-18T12:03:59.061133+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:03:59.061133+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3760745,"num_objects":919,"num_object_clones":0,"num_object_copies":1838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18609,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18722","reported_seq":25871,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:01.147074+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:01.147074+0000","last_peered":"2024-09-18T12:04:01.147074+0000","last_clean":"2024-09-18T12:04:01.147074+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:01.147074+0000","last_undegraded":"2024-09-18T12:04:01.147074+0000","last_fullsized":"2024-09-18T12:04:01.147074+0000","mapping_epoch":40,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3278651,"num_objects":803,"num_object_clones":0,"num_object_copies":1606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18719,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18356","reported_seq":18440,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.044520+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:03:59.044520+0000","last_peered":"2024-09-18T12:03:59.044520+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:03:59.044520+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8300","ondisk_log_start":"39'8300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3450066,"num_objects":844,"num_object_clones":0,"num_object_copies":1688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18354,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18664","reported_seq":18753,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:03:59.086934+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:03:59.086934+0000","last_peered":"2024-09-18T12:03:59.086934+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:03:59.086934+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3475876,"num_objects":852,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18661,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":56467532,"num_objects":13804,"num_object_clones":0,"num_object_copies":27608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":281411,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12709,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":285179904,"data_stored":285116096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051705,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456900,"kb_used_data":142396,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93914940,"statfs":{"total":96636764160,"available":96168898560,"internally_reserved":0,"allocated":145813504,"data_stored":145564869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.30299999999999999},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.27700000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.26400000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855468,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350872,"kb_used_data":40528,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94020968,"statfs":{"total":96636764160,"available":96277471232,"internally_reserved":0,"allocated":41500672,"data_stored":41203188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":2.4700000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":2.3090000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":405636,"kb_used_data":99512,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93966204,"statfs":{"total":96636764160,"available":96221392896,"internally_reserved":0,"allocated":101900288,"data_stored":101652103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,3,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.2440000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100278272,"data_stored":100274793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40034304,"data_stored":39981526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":144867328,"data_stored":144859777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:07.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:08.127 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:08.170 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:08.182 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-18T12:04:07.253723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47534175,"num_objects":11625,"num_object_clones":0,"num_object_copies":23250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11625,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":283594,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12798,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160602,"ondisk_log_size":160602,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1161456,"kb_used_data":263752,"kb_used_omap":13,"kb_used_meta":897650,"kb_avail":281954064,"statfs":{"total":289910292480,"available":288720961536,"internally_reserved":0,"allocated":270082048,"data_stored":269287744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":919194130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,5,16,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18804736,"num_objects":-4591,"num_object_clones":0,"num_object_copies":-9182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002723"},"pg_stats":[{"pgid":"2.f","version":"43'18844","reported_seq":18919,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.154783+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:04.154783+0000","last_peered":"2024-09-18T12:04:04.154783+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:04.154783+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2834432,"num_objects":692,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18844,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2907}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18701","reported_seq":18863,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:03.922539+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:03.922539+0000","last_peered":"2024-09-18T12:04:03.922539+0000","last_clean":"2024-09-18T12:04:03.922539+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:03.922539+0000","last_undegraded":"2024-09-18T12:04:03.922539+0000","last_fullsized":"2024-09-18T12:04:03.922539+0000","mapping_epoch":29,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2994176,"num_objects":731,"num_object_clones":0,"num_object_copies":1462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15893,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18812","reported_seq":19002,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:03.966801+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:03.966801+0000","last_peered":"2024-09-18T12:04:03.966801+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:03.966801+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3051544,"num_objects":746,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":746,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15670,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2790}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18528","reported_seq":18561,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:03.817875+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:03.817875+0000","last_peered":"2024-09-18T12:04:03.817875+0000","last_clean":"2024-09-18T12:04:03.817875+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:03.817875+0000","last_undegraded":"2024-09-18T12:04:03.817875+0000","last_fullsized":"2024-09-18T12:04:03.817875+0000","mapping_epoch":29,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998272,"num_objects":732,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18528,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18715","reported_seq":18910,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:04.182887+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:04.182887+0000","last_peered":"2024-09-18T12:04:04.182887+0000","last_clean":"2024-09-18T12:04:04.182887+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:04.182887+0000","last_undegraded":"2024-09-18T12:04:04.182887+0000","last_fullsized":"2024-09-18T12:04:04.182887+0000","mapping_epoch":29,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998272,"num_objects":732,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18670","reported_seq":18753,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.113883+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:04.113883+0000","last_peered":"2024-09-18T12:04:04.113883+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:04.113883+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3085522,"num_objects":755,"num_object_clones":0,"num_object_copies":1510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18668,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18641","reported_seq":18732,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.086418+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:04.086418+0000","last_peered":"2024-09-18T12:04:04.086418+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:04.086418+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2995410,"num_objects":733,"num_object_clones":0,"num_object_copies":1466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18639,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2826}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18914","reported_seq":19143,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:04.122377+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:04.122377+0000","last_peered":"2024-09-18T12:04:04.122377+0000","last_clean":"2024-09-18T12:04:04.122377+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:04.122377+0000","last_undegraded":"2024-09-18T12:04:04.122377+0000","last_fullsized":"2024-09-18T12:04:04.122377+0000","mapping_epoch":29,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":688,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15231,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18676","reported_seq":25808,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:06.335385+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:06.335385+0000","last_peered":"2024-09-18T12:04:06.335385+0000","last_clean":"2024-09-18T12:04:06.335385+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:06.335385+0000","last_undegraded":"2024-09-18T12:04:06.335385+0000","last_fullsized":"2024-09-18T12:04:06.335385+0000","mapping_epoch":38,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2987218,"num_objects":731,"num_object_clones":0,"num_object_copies":1462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18674,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18723","reported_seq":23150,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.043283+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:04.043283+0000","last_peered":"2024-09-18T12:04:04.043283+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:04.043283+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2962642,"num_objects":725,"num_object_clones":0,"num_object_copies":1450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18721,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-636}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18691","reported_seq":18751,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:03.894061+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:03.894061+0000","last_peered":"2024-09-18T12:04:03.894061+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:03.894061+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3163346,"num_objects":774,"num_object_clones":0,"num_object_copies":1548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18689,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18748","reported_seq":26160,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:03.909381+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:03.909381+0000","last_peered":"2024-09-18T12:04:03.909381+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:03.909381+0000","last_undegraded":"2024-09-18T12:04:03.909381+0000","last_fullsized":"2024-09-18T12:04:03.909381+0000","mapping_epoch":42,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3072000,"num_objects":750,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15228,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":750}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18757","reported_seq":18843,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.107661+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:04.107661+0000","last_peered":"2024-09-18T12:04:04.107661+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:04.107661+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3158633,"num_objects":772,"num_object_clones":0,"num_object_copies":1544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18756,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18864","reported_seq":26010,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:06.331830+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:06.331830+0000","last_peered":"2024-09-18T12:04:06.331830+0000","last_clean":"2024-09-18T12:04:06.331830+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:06.331830+0000","last_undegraded":"2024-09-18T12:04:06.331830+0000","last_fullsized":"2024-09-18T12:04:06.331830+0000","mapping_epoch":40,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2697019,"num_objects":661,"num_object_clones":0,"num_object_copies":1322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18861,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18508","reported_seq":18593,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.174982+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:04.174982+0000","last_peered":"2024-09-18T12:04:04.174982+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:04.174982+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2827474,"num_objects":692,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18506,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2901}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18807","reported_seq":18896,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.145786+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:04.145786+0000","last_peered":"2024-09-18T12:04:04.145786+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:04.145786+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2890148,"num_objects":709,"num_object_clones":0,"num_object_copies":1418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18804,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":47534156,"num_objects":11623,"num_object_clones":0,"num_object_copies":23246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":283592,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12797,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":267124736,"data_stored":267060928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160599,"ondisk_log_size":160599,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051706,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440184,"kb_used_data":138160,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93931656,"statfs":{"total":96636764160,"available":96186015744,"internally_reserved":0,"allocated":141475840,"data_stored":141227205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.25},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":4.2350000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":4.274},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":3.9249999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855469,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331864,"kb_used_data":33996,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94039976,"statfs":{"total":96636764160,"available":96296935424,"internally_reserved":0,"allocated":34811904,"data_stored":34514420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.28799999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389408,"kb_used_data":91596,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93982432,"statfs":{"total":96636764160,"available":96238010368,"internally_reserved":0,"allocated":93794304,"data_stored":93546119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.27600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":93413376,"data_stored":93409897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":33116160,"data_stored":33063382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140595200,"data_stored":140587649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:08.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:08.839 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:08.858 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:08.868 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-18T12:04:07.253723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47534175,"num_objects":11625,"num_object_clones":0,"num_object_copies":23250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11625,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":283594,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12798,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160602,"ondisk_log_size":160602,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1161456,"kb_used_data":263752,"kb_used_omap":13,"kb_used_meta":897650,"kb_avail":281954064,"statfs":{"total":289910292480,"available":288720961536,"internally_reserved":0,"allocated":270082048,"data_stored":269287744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":919194130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,5,16,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18804736,"num_objects":-4591,"num_object_clones":0,"num_object_copies":-9182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4591,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002723"},"pg_stats":[{"pgid":"2.f","version":"43'18844","reported_seq":18919,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.154783+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:04.154783+0000","last_peered":"2024-09-18T12:04:04.154783+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:04.154783+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2834432,"num_objects":692,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18844,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2907}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18701","reported_seq":18863,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:03.922539+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:03.922539+0000","last_peered":"2024-09-18T12:04:03.922539+0000","last_clean":"2024-09-18T12:04:03.922539+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:03.922539+0000","last_undegraded":"2024-09-18T12:04:03.922539+0000","last_fullsized":"2024-09-18T12:04:03.922539+0000","mapping_epoch":29,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2994176,"num_objects":731,"num_object_clones":0,"num_object_copies":1462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15893,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18812","reported_seq":19002,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:03.966801+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:03.966801+0000","last_peered":"2024-09-18T12:04:03.966801+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:03.966801+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3051544,"num_objects":746,"num_object_clones":0,"num_object_copies":1492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":746,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15670,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2790}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18528","reported_seq":18561,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:03.817875+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:03.817875+0000","last_peered":"2024-09-18T12:04:03.817875+0000","last_clean":"2024-09-18T12:04:03.817875+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:03.817875+0000","last_undegraded":"2024-09-18T12:04:03.817875+0000","last_fullsized":"2024-09-18T12:04:03.817875+0000","mapping_epoch":29,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998272,"num_objects":732,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18528,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18715","reported_seq":18910,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:04.182887+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:04.182887+0000","last_peered":"2024-09-18T12:04:04.182887+0000","last_clean":"2024-09-18T12:04:04.182887+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:04.182887+0000","last_undegraded":"2024-09-18T12:04:04.182887+0000","last_fullsized":"2024-09-18T12:04:04.182887+0000","mapping_epoch":29,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998272,"num_objects":732,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18670","reported_seq":18753,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.113883+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:04.113883+0000","last_peered":"2024-09-18T12:04:04.113883+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:04.113883+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3085522,"num_objects":755,"num_object_clones":0,"num_object_copies":1510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18668,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18641","reported_seq":18732,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.086418+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:04.086418+0000","last_peered":"2024-09-18T12:04:04.086418+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:04.086418+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2995410,"num_objects":733,"num_object_clones":0,"num_object_copies":1466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18639,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2826}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18914","reported_seq":19143,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:04.122377+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:04.122377+0000","last_peered":"2024-09-18T12:04:04.122377+0000","last_clean":"2024-09-18T12:04:04.122377+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:04.122377+0000","last_undegraded":"2024-09-18T12:04:04.122377+0000","last_fullsized":"2024-09-18T12:04:04.122377+0000","mapping_epoch":29,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":688,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15231,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18676","reported_seq":25808,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:06.335385+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:06.335385+0000","last_peered":"2024-09-18T12:04:06.335385+0000","last_clean":"2024-09-18T12:04:06.335385+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:06.335385+0000","last_undegraded":"2024-09-18T12:04:06.335385+0000","last_fullsized":"2024-09-18T12:04:06.335385+0000","mapping_epoch":38,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2987218,"num_objects":731,"num_object_clones":0,"num_object_copies":1462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18674,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18723","reported_seq":23150,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.043283+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:04.043283+0000","last_peered":"2024-09-18T12:04:04.043283+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:04.043283+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2962642,"num_objects":725,"num_object_clones":0,"num_object_copies":1450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18721,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-636}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18691","reported_seq":18751,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:03.894061+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:03.894061+0000","last_peered":"2024-09-18T12:04:03.894061+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:03.894061+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3163346,"num_objects":774,"num_object_clones":0,"num_object_copies":1548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18689,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18748","reported_seq":26160,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:03.909381+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:03.909381+0000","last_peered":"2024-09-18T12:04:03.909381+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:03.909381+0000","last_undegraded":"2024-09-18T12:04:03.909381+0000","last_fullsized":"2024-09-18T12:04:03.909381+0000","mapping_epoch":42,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3072000,"num_objects":750,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15228,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":750}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18757","reported_seq":18843,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.107661+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:04.107661+0000","last_peered":"2024-09-18T12:04:04.107661+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:04.107661+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3158633,"num_objects":772,"num_object_clones":0,"num_object_copies":1544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18756,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18864","reported_seq":26010,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:06.331830+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:06.331830+0000","last_peered":"2024-09-18T12:04:06.331830+0000","last_clean":"2024-09-18T12:04:06.331830+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:06.331830+0000","last_undegraded":"2024-09-18T12:04:06.331830+0000","last_fullsized":"2024-09-18T12:04:06.331830+0000","mapping_epoch":40,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2697019,"num_objects":661,"num_object_clones":0,"num_object_copies":1322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18861,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18508","reported_seq":18593,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.174982+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:04.174982+0000","last_peered":"2024-09-18T12:04:04.174982+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:04.174982+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2827474,"num_objects":692,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18506,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2901}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18807","reported_seq":18896,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.145786+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:04.145786+0000","last_peered":"2024-09-18T12:04:04.145786+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:04.145786+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2890148,"num_objects":709,"num_object_clones":0,"num_object_copies":1418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18804,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":47534156,"num_objects":11623,"num_object_clones":0,"num_object_copies":23246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":283592,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12797,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":267124736,"data_stored":267060928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160599,"ondisk_log_size":160599,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051706,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440184,"kb_used_data":138160,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93931656,"statfs":{"total":96636764160,"available":96186015744,"internally_reserved":0,"allocated":141475840,"data_stored":141227205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.25},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":4.2350000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":4.274},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":3.9249999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855469,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331864,"kb_used_data":33996,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94039976,"statfs":{"total":96636764160,"available":96296935424,"internally_reserved":0,"allocated":34811904,"data_stored":34514420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.28799999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389408,"kb_used_data":91596,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93982432,"statfs":{"total":96636764160,"available":96238010368,"internally_reserved":0,"allocated":93794304,"data_stored":93546119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.27600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":93413376,"data_stored":93409897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":33116160,"data_stored":33063382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140595200,"data_stored":140587649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:08.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:04:08.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'18844", 'reported_seq': 18919, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:04.154783+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:04.154783+0000', 'last_peered': '2024-09-18T12:04:04.154783+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:04.154783+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'8800", 'ondisk_log_start': "39'8800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2834432, 'num_objects': 692, 'num_object_clones': 0, 'num_object_copies': 1384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18844, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2907}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:08.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:04:08.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'18812", 'reported_seq': 19002, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:03.966801+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:03.966801+0000', 'last_peered': '2024-09-18T12:04:03.966801+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:03.966801+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'8800", 'ondisk_log_start': "39'8800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3051544, 'num_objects': 746, 'num_object_clones': 0, 'num_object_copies': 1492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 746, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15670, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2790}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:08.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:04:08.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'18670", 'reported_seq': 18753, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:04.113883+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:04.113883+0000', 'last_peered': '2024-09-18T12:04:04.113883+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:04.113883+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'8600", 'ondisk_log_start': "39'8600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3085522, 'num_objects': 755, 'num_object_clones': 0, 'num_object_copies': 1510, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18668, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2772}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:08.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:04:08.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'18641", 'reported_seq': 18732, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:04.086418+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:04.086418+0000', 'last_peered': '2024-09-18T12:04:04.086418+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:04.086418+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'8600", 'ondisk_log_start': "39'8600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2995410, 'num_objects': 733, 'num_object_clones': 0, 'num_object_copies': 1466, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18639, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2826}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:08.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:04:08.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'18723", 'reported_seq': 23150, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:04.043283+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:04:04.043283+0000', 'last_peered': '2024-09-18T12:04:04.043283+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:04:04.043283+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'8700", 'ondisk_log_start': "39'8700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2962642, 'num_objects': 725, 'num_object_clones': 0, 'num_object_copies': 1450, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1361, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18721, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2175, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -636}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:08.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:04:08.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'18691", 'reported_seq': 18751, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:03.894061+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:04:03.894061+0000', 'last_peered': '2024-09-18T12:04:03.894061+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:04:03.894061+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'8600", 'ondisk_log_start': "39'8600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3163346, 'num_objects': 774, 'num_object_clones': 0, 'num_object_copies': 1548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18689, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2776}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:08.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:04:08.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'18748", 'reported_seq': 26160, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:03.909381+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:03.909381+0000', 'last_peered': '2024-09-18T12:04:03.909381+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:03.909381+0000', 'last_undegraded': '2024-09-18T12:04:03.909381+0000', 'last_fullsized': '2024-09-18T12:04:03.909381+0000', 'mapping_epoch': 42, 'log_start': "39'8700", 'ondisk_log_start': "39'8700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3072000, 'num_objects': 750, 'num_object_clones': 0, 'num_object_copies': 1500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15228, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 750}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:08.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:04:08.871 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'18757", 'reported_seq': 18843, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:04.107661+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:04.107661+0000', 'last_peered': '2024-09-18T12:04:04.107661+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:04.107661+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'8700", 'ondisk_log_start': "39'8700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3158633, 'num_objects': 772, 'num_object_clones': 0, 'num_object_copies': 1544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18756, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2763}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:08.872 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:04:08.872 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'18508", 'reported_seq': 18593, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:04.174982+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:04.174982+0000', 'last_peered': '2024-09-18T12:04:04.174982+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:04.174982+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'8500", 'ondisk_log_start': "39'8500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2827474, 'num_objects': 692, 'num_object_clones': 0, 'num_object_copies': 1384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18506, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2901}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:08.872 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:04:08.872 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'18807", 'reported_seq': 18896, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:04.145786+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:04.145786+0000', 'last_peered': '2024-09-18T12:04:04.145786+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:04.145786+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'8800", 'ondisk_log_start': "39'8800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2890148, 'num_objects': 709, 'num_object_clones': 0, 'num_object_copies': 1418, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18804, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2814}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:08.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:04:09.720 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:09.720 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29796/23250 objects degraded (128.155%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":649,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":11625,"data_bytes":47534175,"bytes_used":1189330944,"bytes_avail":288720961536,"bytes_total":289910292480,"degraded_objects":29796,"degraded_total":23250,"degraded_ratio":1.2815483870967741,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":382},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:04:09.745 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:04:09.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:10.433 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:10.442 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:10.483 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-18T12:04:09.254150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":44609631,"num_objects":10911,"num_object_clones":0,"num_object_copies":21822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10911,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":284308,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12798,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160716,"ondisk_log_size":160716,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1161176,"kb_used_data":259312,"kb_used_omap":13,"kb_used_meta":901810,"kb_avail":281954344,"statfs":{"total":289910292480,"available":288721248256,"internally_reserved":0,"allocated":265535488,"data_stored":264741184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":923453970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,4,14,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20627456,"num_objects":-5036,"num_object_clones":0,"num_object_copies":-10072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5036,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002644"},"pg_stats":[{"pgid":"2.f","version":"43'18844","reported_seq":18919,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.154783+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:04.154783+0000","last_peered":"2024-09-18T12:04:04.154783+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:04.154783+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2834432,"num_objects":692,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18844,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2907}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18844","reported_seq":19006,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:08.987652+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:08.987652+0000","last_peered":"2024-09-18T12:04:08.987652+0000","last_clean":"2024-09-18T12:04:08.987652+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:08.987652+0000","last_undegraded":"2024-09-18T12:04:08.987652+0000","last_fullsized":"2024-09-18T12:04:08.987652+0000","mapping_epoch":29,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2408448,"num_objects":588,"num_object_clones":0,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16036,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18954","reported_seq":19144,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:08.928839+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:08.928839+0000","last_peered":"2024-09-18T12:04:08.928839+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:08.928839+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2469912,"num_objects":604,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15812,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2932}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18673","reported_seq":18706,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:08.999936+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:08.999936+0000","last_peered":"2024-09-18T12:04:08.999936+0000","last_clean":"2024-09-18T12:04:08.999936+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:08.999936+0000","last_undegraded":"2024-09-18T12:04:08.999936+0000","last_fullsized":"2024-09-18T12:04:08.999936+0000","mapping_epoch":29,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2404352,"num_objects":587,"num_object_clones":0,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18673,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18715","reported_seq":18910,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:04.182887+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:04.182887+0000","last_peered":"2024-09-18T12:04:04.182887+0000","last_clean":"2024-09-18T12:04:04.182887+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:04.182887+0000","last_undegraded":"2024-09-18T12:04:04.182887+0000","last_fullsized":"2024-09-18T12:04:04.182887+0000","mapping_epoch":29,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2998272,"num_objects":732,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18670","reported_seq":18753,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.113883+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:04.113883+0000","last_peered":"2024-09-18T12:04:04.113883+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:04.113883+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3085522,"num_objects":755,"num_object_clones":0,"num_object_copies":1510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18668,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18641","reported_seq":18732,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.086418+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:04.086418+0000","last_peered":"2024-09-18T12:04:04.086418+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:04.086418+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2995410,"num_objects":733,"num_object_clones":0,"num_object_copies":1466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18639,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2826}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'18914","reported_seq":19143,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:04.122377+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:04.122377+0000","last_peered":"2024-09-18T12:04:04.122377+0000","last_clean":"2024-09-18T12:04:04.122377+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:04.122377+0000","last_undegraded":"2024-09-18T12:04:04.122377+0000","last_fullsized":"2024-09-18T12:04:04.122377+0000","mapping_epoch":29,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":688,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15231,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18676","reported_seq":25808,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:06.335385+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:06.335385+0000","last_peered":"2024-09-18T12:04:06.335385+0000","last_clean":"2024-09-18T12:04:06.335385+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:06.335385+0000","last_undegraded":"2024-09-18T12:04:06.335385+0000","last_fullsized":"2024-09-18T12:04:06.335385+0000","mapping_epoch":38,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2987218,"num_objects":731,"num_object_clones":0,"num_object_copies":1462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18674,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18723","reported_seq":23150,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.043283+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:04.043283+0000","last_peered":"2024-09-18T12:04:04.043283+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:04.043283+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2962642,"num_objects":725,"num_object_clones":0,"num_object_copies":1450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18721,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-636}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18835","reported_seq":18895,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:08.953155+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:08.953155+0000","last_peered":"2024-09-18T12:04:08.953155+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:08.953155+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2573522,"num_objects":630,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18833,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2920}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18888","reported_seq":26301,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:08.977236+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:08.977236+0000","last_peered":"2024-09-18T12:04:08.977236+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:08.977236+0000","last_undegraded":"2024-09-18T12:04:08.977236+0000","last_fullsized":"2024-09-18T12:04:08.977236+0000","mapping_epoch":42,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2498560,"num_objects":610,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15368,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":610}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18757","reported_seq":18843,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.107661+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:04.107661+0000","last_peered":"2024-09-18T12:04:04.107661+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:04.107661+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3158633,"num_objects":772,"num_object_clones":0,"num_object_copies":1544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18756,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2763}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18864","reported_seq":26010,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:06.331830+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:06.331830+0000","last_peered":"2024-09-18T12:04:06.331830+0000","last_clean":"2024-09-18T12:04:06.331830+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:06.331830+0000","last_undegraded":"2024-09-18T12:04:06.331830+0000","last_fullsized":"2024-09-18T12:04:06.331830+0000","mapping_epoch":40,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2697019,"num_objects":661,"num_object_clones":0,"num_object_copies":1322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18861,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18508","reported_seq":18593,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.174982+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:04.174982+0000","last_peered":"2024-09-18T12:04:04.174982+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:04.174982+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8500","ondisk_log_start":"39'8500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2827474,"num_objects":692,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18506,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2901}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18807","reported_seq":18896,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:04.145786+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:04.145786+0000","last_peered":"2024-09-18T12:04:04.145786+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:04.145786+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2890148,"num_objects":709,"num_object_clones":0,"num_object_copies":1418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18804,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":44609612,"num_objects":10909,"num_object_clones":0,"num_object_copies":21818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10909,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":284306,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12797,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262529024,"data_stored":262465216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160713,"ondisk_log_size":160713,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051707,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439904,"kb_used_data":133720,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93931936,"statfs":{"total":96636764160,"available":96186302464,"internally_reserved":0,"allocated":136929280,"data_stored":136680645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":3.589},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.24299999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.27300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855469,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331864,"kb_used_data":33996,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94039976,"statfs":{"total":96636764160,"available":96296935424,"internally_reserved":0,"allocated":34811904,"data_stored":34514420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.28799999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389408,"kb_used_data":91596,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93982432,"statfs":{"total":96636764160,"available":96238010368,"internally_reserved":0,"allocated":93794304,"data_stored":93546119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.27600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":93413376,"data_stored":93409897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":33116160,"data_stored":33063382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":135999488,"data_stored":135991937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:13.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:14.125 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:14.154 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-09-18T12:04:13.255131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38375519,"num_objects":9389,"num_object_clones":0,"num_object_copies":18778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9389,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285830,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12884,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160838,"ondisk_log_size":160838,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1159312,"kb_used_data":244980,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281956208,"statfs":{"total":289910292480,"available":288723156992,"internally_reserved":0,"allocated":250859520,"data_stored":250065216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,11,4,6,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18092032,"num_objects":-4417,"num_object_clones":0,"num_object_copies":-8834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4417,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002680"},"pg_stats":[{"pgid":"2.f","version":"43'18987","reported_seq":19062,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.197262+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:09.197262+0000","last_peered":"2024-09-18T12:04:09.197262+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:09.197262+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2248704,"num_objects":549,"num_object_clones":0,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18987,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3050}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18844","reported_seq":19006,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:08.987652+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:08.987652+0000","last_peered":"2024-09-18T12:04:08.987652+0000","last_clean":"2024-09-18T12:04:08.987652+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:08.987652+0000","last_undegraded":"2024-09-18T12:04:08.987652+0000","last_fullsized":"2024-09-18T12:04:08.987652+0000","mapping_epoch":29,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2408448,"num_objects":588,"num_object_clones":0,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16036,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18954","reported_seq":19144,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:08.928839+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:08.928839+0000","last_peered":"2024-09-18T12:04:08.928839+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:08.928839+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2469912,"num_objects":604,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15812,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2932}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18673","reported_seq":18706,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:08.999936+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:08.999936+0000","last_peered":"2024-09-18T12:04:08.999936+0000","last_clean":"2024-09-18T12:04:08.999936+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:08.999936+0000","last_undegraded":"2024-09-18T12:04:08.999936+0000","last_fullsized":"2024-09-18T12:04:08.999936+0000","mapping_epoch":29,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2404352,"num_objects":587,"num_object_clones":0,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18673,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18856","reported_seq":19051,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:09.184524+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:09.184524+0000","last_peered":"2024-09-18T12:04:09.184524+0000","last_clean":"2024-09-18T12:04:09.184524+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:09.184524+0000","last_undegraded":"2024-09-18T12:04:09.184524+0000","last_fullsized":"2024-09-18T12:04:09.184524+0000","mapping_epoch":29,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2420736,"num_objects":591,"num_object_clones":0,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16021,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18819","reported_seq":18904,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.212920+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:09.212920+0000","last_peered":"2024-09-18T12:04:09.212920+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:09.212920+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2475218,"num_objects":606,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18817,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2921}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18772","reported_seq":18861,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.204896+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:09.204896+0000","last_peered":"2024-09-18T12:04:09.204896+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:09.204896+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2458834,"num_objects":602,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18770,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19052","reported_seq":19281,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:09.227683+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:09.227683+0000","last_peered":"2024-09-18T12:04:09.227683+0000","last_clean":"2024-09-18T12:04:09.227683+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:09.227683+0000","last_undegraded":"2024-09-18T12:04:09.227683+0000","last_fullsized":"2024-09-18T12:04:09.227683+0000","mapping_epoch":29,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252800,"num_objects":550,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15369,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18836","reported_seq":25968,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:11.304401+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:11.304401+0000","last_peered":"2024-09-18T12:04:11.304401+0000","last_clean":"2024-09-18T12:04:11.304401+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:11.304401+0000","last_undegraded":"2024-09-18T12:04:11.304401+0000","last_fullsized":"2024-09-18T12:04:11.304401+0000","mapping_epoch":38,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2331858,"num_objects":571,"num_object_clones":0,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18834,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18855","reported_seq":23454,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.255790+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:09.255790+0000","last_peered":"2024-09-18T12:04:09.255790+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:09.255790+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2421970,"num_objects":593,"num_object_clones":0,"num_object_copies":1186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18853,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2261,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18835","reported_seq":18895,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:08.953155+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:08.953155+0000","last_peered":"2024-09-18T12:04:08.953155+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:08.953155+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2573522,"num_objects":630,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18833,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2920}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18888","reported_seq":26301,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:08.977236+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:08.977236+0000","last_peered":"2024-09-18T12:04:08.977236+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:08.977236+0000","last_undegraded":"2024-09-18T12:04:08.977236+0000","last_fullsized":"2024-09-18T12:04:08.977236+0000","mapping_epoch":42,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2498560,"num_objects":610,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15368,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":610}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18911","reported_seq":18997,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.196302+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:09.196302+0000","last_peered":"2024-09-18T12:04:09.196302+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:09.196302+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2527849,"num_objects":618,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18910,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18989","reported_seq":26139,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:11.273892+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:11.273892+0000","last_peered":"2024-09-18T12:04:11.273892+0000","last_clean":"2024-09-18T12:04:11.273892+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:11.273892+0000","last_undegraded":"2024-09-18T12:04:11.273892+0000","last_fullsized":"2024-09-18T12:04:11.273892+0000","mapping_epoch":40,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185019,"num_objects":536,"num_object_clones":0,"num_object_copies":1072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18986,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18629","reported_seq":18714,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.063138+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:09.063138+0000","last_peered":"2024-09-18T12:04:09.063138+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:09.063138+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2331858,"num_objects":571,"num_object_clones":0,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18627,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3022}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18935","reported_seq":19024,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.143734+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:09.143734+0000","last_peered":"2024-09-18T12:04:09.143734+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:09.143734+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2365860,"num_objects":581,"num_object_clones":0,"num_object_copies":1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18932,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2942}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":38375500,"num_objects":9387,"num_object_clones":0,"num_object_copies":18774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285828,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12883,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249008128,"data_stored":248944320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160835,"ondisk_log_size":160835,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051707,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439904,"kb_used_data":133720,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93931936,"statfs":{"total":96636764160,"available":96186302464,"internally_reserved":0,"allocated":136929280,"data_stored":136680645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":3.589},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.24299999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.27300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855470,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332476,"kb_used_data":26292,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94039364,"statfs":{"total":96636764160,"available":96296308736,"internally_reserved":0,"allocated":26923008,"data_stored":26625524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.24299999999999999},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.39200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.28999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386932,"kb_used_data":84968,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93984908,"statfs":{"total":96636764160,"available":96240545792,"internally_reserved":0,"allocated":87007232,"data_stored":86759047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,6,0,3,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":5.1749999999999998},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.249}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":8.8650000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":5.21}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86614016,"data_stored":86610537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26394624,"data_stored":26341846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":135999488,"data_stored":135991937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:14.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:14.958 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:14.966 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:14.977 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-09-18T12:04:13.255131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38375519,"num_objects":9389,"num_object_clones":0,"num_object_copies":18778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9389,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285830,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12884,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160838,"ondisk_log_size":160838,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1159312,"kb_used_data":244980,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":281956208,"statfs":{"total":289910292480,"available":288723156992,"internally_reserved":0,"allocated":250859520,"data_stored":250065216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,11,4,6,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18092032,"num_objects":-4417,"num_object_clones":0,"num_object_copies":-8834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4417,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002680"},"pg_stats":[{"pgid":"2.f","version":"43'18987","reported_seq":19062,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.197262+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:09.197262+0000","last_peered":"2024-09-18T12:04:09.197262+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:09.197262+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2248704,"num_objects":549,"num_object_clones":0,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18987,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3050}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18844","reported_seq":19006,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:08.987652+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:08.987652+0000","last_peered":"2024-09-18T12:04:08.987652+0000","last_clean":"2024-09-18T12:04:08.987652+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:08.987652+0000","last_undegraded":"2024-09-18T12:04:08.987652+0000","last_fullsized":"2024-09-18T12:04:08.987652+0000","mapping_epoch":29,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2408448,"num_objects":588,"num_object_clones":0,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16036,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'18954","reported_seq":19144,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:08.928839+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:08.928839+0000","last_peered":"2024-09-18T12:04:08.928839+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:08.928839+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2469912,"num_objects":604,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15812,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2932}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18673","reported_seq":18706,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:08.999936+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:08.999936+0000","last_peered":"2024-09-18T12:04:08.999936+0000","last_clean":"2024-09-18T12:04:08.999936+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:08.999936+0000","last_undegraded":"2024-09-18T12:04:08.999936+0000","last_fullsized":"2024-09-18T12:04:08.999936+0000","mapping_epoch":29,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2404352,"num_objects":587,"num_object_clones":0,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18673,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'18856","reported_seq":19051,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:09.184524+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:09.184524+0000","last_peered":"2024-09-18T12:04:09.184524+0000","last_clean":"2024-09-18T12:04:09.184524+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:09.184524+0000","last_undegraded":"2024-09-18T12:04:09.184524+0000","last_fullsized":"2024-09-18T12:04:09.184524+0000","mapping_epoch":29,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2420736,"num_objects":591,"num_object_clones":0,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16021,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18819","reported_seq":18904,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.212920+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:09.212920+0000","last_peered":"2024-09-18T12:04:09.212920+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:09.212920+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2475218,"num_objects":606,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18817,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2921}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18772","reported_seq":18861,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.204896+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:09.204896+0000","last_peered":"2024-09-18T12:04:09.204896+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:09.204896+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2458834,"num_objects":602,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18770,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19052","reported_seq":19281,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:09.227683+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:09.227683+0000","last_peered":"2024-09-18T12:04:09.227683+0000","last_clean":"2024-09-18T12:04:09.227683+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:09.227683+0000","last_undegraded":"2024-09-18T12:04:09.227683+0000","last_fullsized":"2024-09-18T12:04:09.227683+0000","mapping_epoch":29,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252800,"num_objects":550,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15369,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18836","reported_seq":25968,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:11.304401+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:11.304401+0000","last_peered":"2024-09-18T12:04:11.304401+0000","last_clean":"2024-09-18T12:04:11.304401+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:11.304401+0000","last_undegraded":"2024-09-18T12:04:11.304401+0000","last_fullsized":"2024-09-18T12:04:11.304401+0000","mapping_epoch":38,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2331858,"num_objects":571,"num_object_clones":0,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18834,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18855","reported_seq":23454,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.255790+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:09.255790+0000","last_peered":"2024-09-18T12:04:09.255790+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:09.255790+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2421970,"num_objects":593,"num_object_clones":0,"num_object_copies":1186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18853,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2261,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18835","reported_seq":18895,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:08.953155+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:08.953155+0000","last_peered":"2024-09-18T12:04:08.953155+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:08.953155+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2573522,"num_objects":630,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18833,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2920}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'18888","reported_seq":26301,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:08.977236+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:08.977236+0000","last_peered":"2024-09-18T12:04:08.977236+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:08.977236+0000","last_undegraded":"2024-09-18T12:04:08.977236+0000","last_fullsized":"2024-09-18T12:04:08.977236+0000","mapping_epoch":42,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2498560,"num_objects":610,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15368,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":610}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'18911","reported_seq":18997,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.196302+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:09.196302+0000","last_peered":"2024-09-18T12:04:09.196302+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:09.196302+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2527849,"num_objects":618,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18910,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18989","reported_seq":26139,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:11.273892+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:11.273892+0000","last_peered":"2024-09-18T12:04:11.273892+0000","last_clean":"2024-09-18T12:04:11.273892+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:11.273892+0000","last_undegraded":"2024-09-18T12:04:11.273892+0000","last_fullsized":"2024-09-18T12:04:11.273892+0000","mapping_epoch":40,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185019,"num_objects":536,"num_object_clones":0,"num_object_copies":1072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18986,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18629","reported_seq":18714,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.063138+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:09.063138+0000","last_peered":"2024-09-18T12:04:09.063138+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:09.063138+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8600","ondisk_log_start":"39'8600","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2331858,"num_objects":571,"num_object_clones":0,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18627,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3022}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'18935","reported_seq":19024,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:09.143734+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:09.143734+0000","last_peered":"2024-09-18T12:04:09.143734+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:09.143734+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2365860,"num_objects":581,"num_object_clones":0,"num_object_copies":1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18932,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2942}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":38375500,"num_objects":9387,"num_object_clones":0,"num_object_copies":18774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285828,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12883,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249008128,"data_stored":248944320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160835,"ondisk_log_size":160835,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051707,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439904,"kb_used_data":133720,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93931936,"statfs":{"total":96636764160,"available":96186302464,"internally_reserved":0,"allocated":136929280,"data_stored":136680645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":3.589},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.24299999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.27300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855470,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332476,"kb_used_data":26292,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94039364,"statfs":{"total":96636764160,"available":96296308736,"internally_reserved":0,"allocated":26923008,"data_stored":26625524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.24299999999999999},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.39200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.28999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386932,"kb_used_data":84968,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93984908,"statfs":{"total":96636764160,"available":96240545792,"internally_reserved":0,"allocated":87007232,"data_stored":86759047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,6,0,3,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":5.1749999999999998},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.249}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":8.8650000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":5.21}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86614016,"data_stored":86610537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26394624,"data_stored":26341846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":135999488,"data_stored":135991937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:14.978 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:04:14.978 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'18987", 'reported_seq': 19062, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:09.197262+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:09.197262+0000', 'last_peered': '2024-09-18T12:04:09.197262+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:09.197262+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'8900", 'ondisk_log_start': "39'8900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2248704, 'num_objects': 549, 'num_object_clones': 0, 'num_object_copies': 1098, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 549, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18987, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3050}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:14.979 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:04:14.979 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'18954", 'reported_seq': 19144, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:08.928839+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:08.928839+0000', 'last_peered': '2024-09-18T12:04:08.928839+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:08.928839+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'8900", 'ondisk_log_start': "39'8900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2469912, 'num_objects': 604, 'num_object_clones': 0, 'num_object_copies': 1208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 604, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15812, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2932}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:14.979 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:04:14.979 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'18819", 'reported_seq': 18904, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:09.212920+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:09.212920+0000', 'last_peered': '2024-09-18T12:04:09.212920+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:09.212920+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'8800", 'ondisk_log_start': "39'8800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2475218, 'num_objects': 606, 'num_object_clones': 0, 'num_object_copies': 1212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18817, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2921}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:14.979 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:04:14.979 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'18772", 'reported_seq': 18861, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:09.204896+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:09.204896+0000', 'last_peered': '2024-09-18T12:04:09.204896+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:09.204896+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'8700", 'ondisk_log_start': "39'8700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2458834, 'num_objects': 602, 'num_object_clones': 0, 'num_object_copies': 1204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18770, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2957}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:14.979 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:04:14.979 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'18855", 'reported_seq': 23454, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:09.255790+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:04:09.255790+0000', 'last_peered': '2024-09-18T12:04:09.255790+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:04:09.255790+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'8800", 'ondisk_log_start': "39'8800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2421970, 'num_objects': 593, 'num_object_clones': 0, 'num_object_copies': 1186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1275, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18853, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2261, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -682}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:14.979 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:04:14.979 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'18835", 'reported_seq': 18895, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:08.953155+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:04:08.953155+0000', 'last_peered': '2024-09-18T12:04:08.953155+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:04:08.953155+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'8800", 'ondisk_log_start': "39'8800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2573522, 'num_objects': 630, 'num_object_clones': 0, 'num_object_copies': 1260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18833, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2920}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:14.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:04:14.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'18888", 'reported_seq': 26301, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:08.977236+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:08.977236+0000', 'last_peered': '2024-09-18T12:04:08.977236+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:08.977236+0000', 'last_undegraded': '2024-09-18T12:04:08.977236+0000', 'last_fullsized': '2024-09-18T12:04:08.977236+0000', 'mapping_epoch': 42, 'log_start': "39'8800", 'ondisk_log_start': "39'8800", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2498560, 'num_objects': 610, 'num_object_clones': 0, 'num_object_copies': 1220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15368, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 610}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:14.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:04:14.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'18911", 'reported_seq': 18997, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:09.196302+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:09.196302+0000', 'last_peered': '2024-09-18T12:04:09.196302+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:09.196302+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'8900", 'ondisk_log_start': "39'8900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2527849, 'num_objects': 618, 'num_object_clones': 0, 'num_object_copies': 1236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 618, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18910, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2917}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:14.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:04:14.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'18629", 'reported_seq': 18714, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:09.063138+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:09.063138+0000', 'last_peered': '2024-09-18T12:04:09.063138+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:09.063138+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'8600", 'ondisk_log_start': "39'8600", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2331858, 'num_objects': 571, 'num_object_clones': 0, 'num_object_copies': 1142, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 571, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18627, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3022}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:14.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:04:14.980 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'18935", 'reported_seq': 19024, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:09.143734+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:09.143734+0000', 'last_peered': '2024-09-18T12:04:09.143734+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:09.143734+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'8900", 'ondisk_log_start': "39'8900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2365860, 'num_objects': 581, 'num_object_clones': 0, 'num_object_copies': 1162, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18932, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2942}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:14.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:04:15.740 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:15.741 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29710/18778 objects degraded (158.217%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":655,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":9389,"data_bytes":38375519,"bytes_used":1187135488,"bytes_avail":288723156992,"bytes_total":289910292480,"degraded_objects":29710,"degraded_total":18778,"degraded_ratio":1.5821706251997021,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":368},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:04:15.767 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:04:15.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:16.438 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:16.454 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-09-18T12:04:15.255564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30589023,"num_objects":7488,"num_object_clones":0,"num_object_copies":14976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":287731,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12969,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160939,"ondisk_log_size":160939,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1158036,"kb_used_data":235384,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281957484,"statfs":{"total":289910292480,"available":288724463616,"internally_reserved":0,"allocated":241033216,"data_stored":240238912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,7,10,7,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25878528,"num_objects":-6318,"num_object_clones":0,"num_object_copies":-12636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002662"},"pg_stats":[{"pgid":"2.f","version":"43'19111","reported_seq":19180,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.208231+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:14.208231+0000","last_peered":"2024-09-18T12:04:14.208231+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:14.208231+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1740800,"num_objects":425,"num_object_clones":0,"num_object_copies":850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19111,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3174}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'18971","reported_seq":19133,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:13.905376+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:13.905376+0000","last_peered":"2024-09-18T12:04:13.905376+0000","last_clean":"2024-09-18T12:04:13.905376+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:13.905376+0000","last_undegraded":"2024-09-18T12:04:13.905376+0000","last_fullsized":"2024-09-18T12:04:13.905376+0000","mapping_epoch":29,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1888256,"num_objects":461,"num_object_clones":0,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16163,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19078","reported_seq":19268,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:13.704924+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:13.704924+0000","last_peered":"2024-09-18T12:04:13.704924+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:13.704924+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1962008,"num_objects":480,"num_object_clones":0,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15936,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3056}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18797","reported_seq":18830,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:13.905135+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:13.905135+0000","last_peered":"2024-09-18T12:04:13.905135+0000","last_clean":"2024-09-18T12:04:13.905135+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:13.905135+0000","last_undegraded":"2024-09-18T12:04:13.905135+0000","last_fullsized":"2024-09-18T12:04:13.905135+0000","mapping_epoch":29,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1896448,"num_objects":463,"num_object_clones":0,"num_object_copies":926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18797,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19001","reported_seq":19196,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:14.226243+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:14.226243+0000","last_peered":"2024-09-18T12:04:14.226243+0000","last_clean":"2024-09-18T12:04:14.226243+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:14.226243+0000","last_undegraded":"2024-09-18T12:04:14.226243+0000","last_fullsized":"2024-09-18T12:04:14.226243+0000","mapping_epoch":29,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1826816,"num_objects":446,"num_object_clones":0,"num_object_copies":892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16166,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18961","reported_seq":19046,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.207566+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:14.207566+0000","last_peered":"2024-09-18T12:04:14.207566+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:14.207566+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1893586,"num_objects":464,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18959,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3063}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18906","reported_seq":18997,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.234742+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:14.234742+0000","last_peered":"2024-09-18T12:04:14.234742+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:14.234742+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1909970,"num_objects":468,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18904,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3091}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19184","reported_seq":19413,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:14.214953+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:14.214953+0000","last_peered":"2024-09-18T12:04:14.214953+0000","last_clean":"2024-09-18T12:04:14.214953+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:14.214953+0000","last_undegraded":"2024-09-18T12:04:14.214953+0000","last_fullsized":"2024-09-18T12:04:14.214953+0000","mapping_epoch":29,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712128,"num_objects":418,"num_object_clones":0,"num_object_copies":836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15501,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18836","reported_seq":25968,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:11.304401+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:11.304401+0000","last_peered":"2024-09-18T12:04:11.304401+0000","last_clean":"2024-09-18T12:04:11.304401+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:11.304401+0000","last_undegraded":"2024-09-18T12:04:11.304401+0000","last_fullsized":"2024-09-18T12:04:11.304401+0000","mapping_epoch":38,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2331858,"num_objects":571,"num_object_clones":0,"num_object_copies":1142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18834,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18999","reported_seq":23770,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.232822+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:14.232822+0000","last_peered":"2024-09-18T12:04:14.232822+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:14.232822+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1832146,"num_objects":449,"num_object_clones":0,"num_object_copies":898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18997,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-741}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'18978","reported_seq":19038,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:13.815523+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:13.815523+0000","last_peered":"2024-09-18T12:04:13.815523+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:13.815523+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1987794,"num_objects":487,"num_object_clones":0,"num_object_copies":974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18976,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3063}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19015","reported_seq":26428,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:13.966232+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:13.966232+0000","last_peered":"2024-09-18T12:04:13.966232+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:13.966232+0000","last_undegraded":"2024-09-18T12:04:13.966232+0000","last_fullsized":"2024-09-18T12:04:13.966232+0000","mapping_epoch":42,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1978368,"num_objects":483,"num_object_clones":0,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15495,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19054","reported_seq":19139,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.227669+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:14.227669+0000","last_peered":"2024-09-18T12:04:14.227669+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:14.227669+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1942121,"num_objects":475,"num_object_clones":0,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19053,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3060}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'18989","reported_seq":26139,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:11.273892+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:11.273892+0000","last_peered":"2024-09-18T12:04:11.273892+0000","last_clean":"2024-09-18T12:04:11.273892+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:11.273892+0000","last_undegraded":"2024-09-18T12:04:11.273892+0000","last_fullsized":"2024-09-18T12:04:11.273892+0000","mapping_epoch":40,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185019,"num_objects":536,"num_object_clones":0,"num_object_copies":1072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18986,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18775","reported_seq":18860,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.094929+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:14.094929+0000","last_peered":"2024-09-18T12:04:14.094929+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:14.094929+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1733842,"num_objects":425,"num_object_clones":0,"num_object_copies":850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18773,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19081","reported_seq":19170,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.139063+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:14.139063+0000","last_peered":"2024-09-18T12:04:14.139063+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:14.139063+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1767844,"num_objects":435,"num_object_clones":0,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19078,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":30589004,"num_objects":7486,"num_object_clones":0,"num_object_copies":14972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":287729,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12968,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237903872,"data_stored":237840064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160936,"ondisk_log_size":160936,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051708,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439692,"kb_used_data":129348,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93932148,"statfs":{"total":96636764160,"available":96186519552,"internally_reserved":0,"allocated":132452352,"data_stored":132203717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.27900000000000003},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.29299999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.619,"15min":1.827},"min":{"1min":0.23200000000000001,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":9.1050000000000004,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3500000000000001,"15min":1.0700000000000001},"min":{"1min":0.22700000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.2910000000000004,"5min":12.712,"15min":12.712},"last":0.31}]}]},{"osd":1,"up_from":35,"seq":150323855470,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332476,"kb_used_data":26292,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94039364,"statfs":{"total":96636764160,"available":96296308736,"internally_reserved":0,"allocated":26923008,"data_stored":26625524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.24299999999999999},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.39200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.28999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385868,"kb_used_data":79744,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93985972,"statfs":{"total":96636764160,"available":96241635328,"internally_reserved":0,"allocated":81657856,"data_stored":81409671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,6,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.249},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.161}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.221},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79892480,"data_stored":79889001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26394624,"data_stored":26341846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131616768,"data_stored":131609217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:19.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:20.030 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:20.066 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:20.091 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-09-18T12:04:19.257663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26566751,"num_objects":6506,"num_object_clones":0,"num_object_copies":13012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":288713,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12969,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160721,"ondisk_log_size":160721,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1155612,"kb_used_data":224640,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":281959908,"statfs":{"total":289910292480,"available":288726945792,"internally_reserved":0,"allocated":230031360,"data_stored":229237056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,13,7,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20967424,"num_objects":-5119,"num_object_clones":0,"num_object_copies":-10238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5119,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003940"},"pg_stats":[{"pgid":"2.f","version":"43'19111","reported_seq":19180,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.208231+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:14.208231+0000","last_peered":"2024-09-18T12:04:14.208231+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:14.208231+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1740800,"num_objects":425,"num_object_clones":0,"num_object_copies":850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19111,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3174}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19120","reported_seq":19282,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:18.925195+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:18.925195+0000","last_peered":"2024-09-18T12:04:18.925195+0000","last_clean":"2024-09-18T12:04:18.925195+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:18.925195+0000","last_undegraded":"2024-09-18T12:04:18.925195+0000","last_fullsized":"2024-09-18T12:04:18.925195+0000","mapping_epoch":29,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1277952,"num_objects":312,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16312,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19224","reported_seq":19412,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:18.997427+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:18.997427+0000","last_peered":"2024-09-18T12:04:18.997427+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:18.997427+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363992,"num_objects":334,"num_object_clones":0,"num_object_copies":668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16082,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18927","reported_seq":18960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:18.956078+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:18.956078+0000","last_peered":"2024-09-18T12:04:18.956078+0000","last_clean":"2024-09-18T12:04:18.956078+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:18.956078+0000","last_undegraded":"2024-09-18T12:04:18.956078+0000","last_fullsized":"2024-09-18T12:04:18.956078+0000","mapping_epoch":29,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363968,"num_objects":333,"num_object_clones":0,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18927,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19001","reported_seq":19196,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:14.226243+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:14.226243+0000","last_peered":"2024-09-18T12:04:14.226243+0000","last_clean":"2024-09-18T12:04:14.226243+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:14.226243+0000","last_undegraded":"2024-09-18T12:04:14.226243+0000","last_fullsized":"2024-09-18T12:04:14.226243+0000","mapping_epoch":29,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1826816,"num_objects":446,"num_object_clones":0,"num_object_copies":892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16166,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18961","reported_seq":19046,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.207566+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:14.207566+0000","last_peered":"2024-09-18T12:04:14.207566+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:14.207566+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1893586,"num_objects":464,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18959,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3063}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18906","reported_seq":18997,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.234742+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:14.234742+0000","last_peered":"2024-09-18T12:04:14.234742+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:14.234742+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1909970,"num_objects":468,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18904,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3091}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19184","reported_seq":19413,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:14.214953+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:14.214953+0000","last_peered":"2024-09-18T12:04:14.214953+0000","last_clean":"2024-09-18T12:04:14.214953+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:14.214953+0000","last_undegraded":"2024-09-18T12:04:14.214953+0000","last_fullsized":"2024-09-18T12:04:14.214953+0000","mapping_epoch":29,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712128,"num_objects":418,"num_object_clones":0,"num_object_copies":836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15501,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18985","reported_seq":26115,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:16.375774+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:16.375774+0000","last_peered":"2024-09-18T12:04:16.375774+0000","last_clean":"2024-09-18T12:04:16.375774+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:16.375774+0000","last_undegraded":"2024-09-18T12:04:16.375774+0000","last_fullsized":"2024-09-18T12:04:16.375774+0000","mapping_epoch":38,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1721554,"num_objects":422,"num_object_clones":0,"num_object_copies":844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18983,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18999","reported_seq":23770,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.232822+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:14.232822+0000","last_peered":"2024-09-18T12:04:14.232822+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:14.232822+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1832146,"num_objects":449,"num_object_clones":0,"num_object_copies":898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18997,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-741}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19117","reported_seq":19177,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:18.863178+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:18.863178+0000","last_peered":"2024-09-18T12:04:18.863178+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:18.863178+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1418450,"num_objects":348,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19115,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3202}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19162","reported_seq":26575,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:18.973653+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:18.973653+0000","last_peered":"2024-09-18T12:04:18.973653+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:18.973653+0000","last_undegraded":"2024-09-18T12:04:18.973653+0000","last_fullsized":"2024-09-18T12:04:18.973653+0000","mapping_epoch":42,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":336,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15642,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":336}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19054","reported_seq":19139,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.227669+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:14.227669+0000","last_peered":"2024-09-18T12:04:14.227669+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:14.227669+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1942121,"num_objects":475,"num_object_clones":0,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19053,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3060}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19111","reported_seq":26261,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:16.323703+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:16.323703+0000","last_peered":"2024-09-18T12:04:16.323703+0000","last_clean":"2024-09-18T12:04:16.323703+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:16.323703+0000","last_undegraded":"2024-09-18T12:04:16.323703+0000","last_fullsized":"2024-09-18T12:04:16.323703+0000","mapping_epoch":40,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1685307,"num_objects":414,"num_object_clones":0,"num_object_copies":828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19108,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18775","reported_seq":18860,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.094929+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:14.094929+0000","last_peered":"2024-09-18T12:04:14.094929+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:14.094929+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1733842,"num_objects":425,"num_object_clones":0,"num_object_copies":850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18773,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19081","reported_seq":19170,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.139063+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:14.139063+0000","last_peered":"2024-09-18T12:04:14.139063+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:14.139063+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1767844,"num_objects":435,"num_object_clones":0,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19078,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":26566732,"num_objects":6504,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":288711,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12968,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":226668544,"data_stored":226604736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160718,"ondisk_log_size":160718,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051709,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439376,"kb_used_data":124872,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93932464,"statfs":{"total":96636764160,"available":96186843136,"internally_reserved":0,"allocated":127868928,"data_stored":127620293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.33000000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.27300000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.41699999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855471,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330368,"kb_used_data":20024,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94041472,"statfs":{"total":96636764160,"available":96298467328,"internally_reserved":0,"allocated":20504576,"data_stored":20207092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.216}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.316},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.37}]}]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385868,"kb_used_data":79744,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93985972,"statfs":{"total":96636764160,"available":96241635328,"internally_reserved":0,"allocated":81657856,"data_stored":81409671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,6,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.249},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.161}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.221},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79892480,"data_stored":79889001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19869696,"data_stored":19816918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126906368,"data_stored":126898817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:20.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:20.797 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:20.846 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:20.859 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-09-18T12:04:19.257663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26566751,"num_objects":6506,"num_object_clones":0,"num_object_copies":13012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":288713,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12969,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160721,"ondisk_log_size":160721,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1155612,"kb_used_data":224640,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":281959908,"statfs":{"total":289910292480,"available":288726945792,"internally_reserved":0,"allocated":230031360,"data_stored":229237056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,13,7,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20967424,"num_objects":-5119,"num_object_clones":0,"num_object_copies":-10238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5119,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003940"},"pg_stats":[{"pgid":"2.f","version":"43'19111","reported_seq":19180,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.208231+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:14.208231+0000","last_peered":"2024-09-18T12:04:14.208231+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:14.208231+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1740800,"num_objects":425,"num_object_clones":0,"num_object_copies":850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19111,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3174}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19120","reported_seq":19282,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:18.925195+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:18.925195+0000","last_peered":"2024-09-18T12:04:18.925195+0000","last_clean":"2024-09-18T12:04:18.925195+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:18.925195+0000","last_undegraded":"2024-09-18T12:04:18.925195+0000","last_fullsized":"2024-09-18T12:04:18.925195+0000","mapping_epoch":29,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1277952,"num_objects":312,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16312,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19224","reported_seq":19412,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:18.997427+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:18.997427+0000","last_peered":"2024-09-18T12:04:18.997427+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:18.997427+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363992,"num_objects":334,"num_object_clones":0,"num_object_copies":668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16082,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18927","reported_seq":18960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:18.956078+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:18.956078+0000","last_peered":"2024-09-18T12:04:18.956078+0000","last_clean":"2024-09-18T12:04:18.956078+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:18.956078+0000","last_undegraded":"2024-09-18T12:04:18.956078+0000","last_fullsized":"2024-09-18T12:04:18.956078+0000","mapping_epoch":29,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363968,"num_objects":333,"num_object_clones":0,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18927,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19001","reported_seq":19196,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:14.226243+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:14.226243+0000","last_peered":"2024-09-18T12:04:14.226243+0000","last_clean":"2024-09-18T12:04:14.226243+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:14.226243+0000","last_undegraded":"2024-09-18T12:04:14.226243+0000","last_fullsized":"2024-09-18T12:04:14.226243+0000","mapping_epoch":29,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1826816,"num_objects":446,"num_object_clones":0,"num_object_copies":892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16166,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'18961","reported_seq":19046,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.207566+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:14.207566+0000","last_peered":"2024-09-18T12:04:14.207566+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:14.207566+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1893586,"num_objects":464,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18959,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3063}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'18906","reported_seq":18997,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.234742+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:14.234742+0000","last_peered":"2024-09-18T12:04:14.234742+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:14.234742+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1909970,"num_objects":468,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18904,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3091}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19184","reported_seq":19413,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:14.214953+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:14.214953+0000","last_peered":"2024-09-18T12:04:14.214953+0000","last_clean":"2024-09-18T12:04:14.214953+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:14.214953+0000","last_undegraded":"2024-09-18T12:04:14.214953+0000","last_fullsized":"2024-09-18T12:04:14.214953+0000","mapping_epoch":29,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712128,"num_objects":418,"num_object_clones":0,"num_object_copies":836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15501,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18985","reported_seq":26115,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:16.375774+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:16.375774+0000","last_peered":"2024-09-18T12:04:16.375774+0000","last_clean":"2024-09-18T12:04:16.375774+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:16.375774+0000","last_undegraded":"2024-09-18T12:04:16.375774+0000","last_fullsized":"2024-09-18T12:04:16.375774+0000","mapping_epoch":38,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1721554,"num_objects":422,"num_object_clones":0,"num_object_copies":844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18983,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'18999","reported_seq":23770,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.232822+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:14.232822+0000","last_peered":"2024-09-18T12:04:14.232822+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:14.232822+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1832146,"num_objects":449,"num_object_clones":0,"num_object_copies":898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18997,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-741}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19117","reported_seq":19177,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:18.863178+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:18.863178+0000","last_peered":"2024-09-18T12:04:18.863178+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:18.863178+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1418450,"num_objects":348,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19115,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3202}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19162","reported_seq":26575,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:18.973653+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:18.973653+0000","last_peered":"2024-09-18T12:04:18.973653+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:18.973653+0000","last_undegraded":"2024-09-18T12:04:18.973653+0000","last_fullsized":"2024-09-18T12:04:18.973653+0000","mapping_epoch":42,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":336,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15642,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":336}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19054","reported_seq":19139,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.227669+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:14.227669+0000","last_peered":"2024-09-18T12:04:14.227669+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:14.227669+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1942121,"num_objects":475,"num_object_clones":0,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19053,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3060}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19111","reported_seq":26261,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:16.323703+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:16.323703+0000","last_peered":"2024-09-18T12:04:16.323703+0000","last_clean":"2024-09-18T12:04:16.323703+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:16.323703+0000","last_undegraded":"2024-09-18T12:04:16.323703+0000","last_fullsized":"2024-09-18T12:04:16.323703+0000","mapping_epoch":40,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1685307,"num_objects":414,"num_object_clones":0,"num_object_copies":828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19108,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18775","reported_seq":18860,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.094929+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:14.094929+0000","last_peered":"2024-09-18T12:04:14.094929+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:14.094929+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8700","ondisk_log_start":"39'8700","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1733842,"num_objects":425,"num_object_clones":0,"num_object_copies":850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18773,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19081","reported_seq":19170,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:14.139063+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:14.139063+0000","last_peered":"2024-09-18T12:04:14.139063+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:14.139063+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1767844,"num_objects":435,"num_object_clones":0,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19078,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":26566732,"num_objects":6504,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":288711,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12968,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":226668544,"data_stored":226604736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160718,"ondisk_log_size":160718,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051709,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439376,"kb_used_data":124872,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93932464,"statfs":{"total":96636764160,"available":96186843136,"internally_reserved":0,"allocated":127868928,"data_stored":127620293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.33000000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.27300000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.41699999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855471,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330368,"kb_used_data":20024,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94041472,"statfs":{"total":96636764160,"available":96298467328,"internally_reserved":0,"allocated":20504576,"data_stored":20207092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.216}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.316},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.37}]}]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385868,"kb_used_data":79744,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93985972,"statfs":{"total":96636764160,"available":96241635328,"internally_reserved":0,"allocated":81657856,"data_stored":81409671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313454086},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,6,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.249},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":3.161}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.221},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79892480,"data_stored":79889001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19869696,"data_stored":19816918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126906368,"data_stored":126898817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:20.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:04:20.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19111", 'reported_seq': 19180, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:14.208231+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:14.208231+0000', 'last_peered': '2024-09-18T12:04:14.208231+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:14.208231+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1740800, 'num_objects': 425, 'num_object_clones': 0, 'num_object_copies': 850, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19111, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3174}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:20.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:04:20.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19224", 'reported_seq': 19412, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:18.997427+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:18.997427+0000', 'last_peered': '2024-09-18T12:04:18.997427+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:18.997427+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9200", 'ondisk_log_start': "39'9200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1363992, 'num_objects': 334, 'num_object_clones': 0, 'num_object_copies': 668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 334, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16082, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3202}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:20.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:04:20.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'18961", 'reported_seq': 19046, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:14.207566+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:14.207566+0000', 'last_peered': '2024-09-18T12:04:14.207566+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:14.207566+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'8900", 'ondisk_log_start': "39'8900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1893586, 'num_objects': 464, 'num_object_clones': 0, 'num_object_copies': 928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18959, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3063}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:20.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:04:20.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'18906", 'reported_seq': 18997, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:14.234742+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:14.234742+0000', 'last_peered': '2024-09-18T12:04:14.234742+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:14.234742+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'8900", 'ondisk_log_start': "39'8900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1909970, 'num_objects': 468, 'num_object_clones': 0, 'num_object_copies': 936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18904, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3091}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:20.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:04:20.861 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'18999", 'reported_seq': 23770, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:14.232822+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:04:14.232822+0000', 'last_peered': '2024-09-18T12:04:14.232822+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:04:14.232822+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'8900", 'ondisk_log_start': "39'8900", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1832146, 'num_objects': 449, 'num_object_clones': 0, 'num_object_copies': 898, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1190, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18997, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2346, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -741}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:20.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:04:20.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19117", 'reported_seq': 19177, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:18.863178+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:04:18.863178+0000', 'last_peered': '2024-09-18T12:04:18.863178+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:04:18.863178+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1418450, 'num_objects': 348, 'num_object_clones': 0, 'num_object_copies': 696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19115, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3202}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:20.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:04:20.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19162", 'reported_seq': 26575, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:18.973653+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:18.973653+0000', 'last_peered': '2024-09-18T12:04:18.973653+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:18.973653+0000', 'last_undegraded': '2024-09-18T12:04:18.973653+0000', 'last_fullsized': '2024-09-18T12:04:18.973653+0000', 'mapping_epoch': 42, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1376256, 'num_objects': 336, 'num_object_clones': 0, 'num_object_copies': 672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 336, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15642, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 336}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:20.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:04:20.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19054", 'reported_seq': 19139, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:14.227669+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:14.227669+0000', 'last_peered': '2024-09-18T12:04:14.227669+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:14.227669+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9000", 'ondisk_log_start': "39'9000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1942121, 'num_objects': 475, 'num_object_clones': 0, 'num_object_copies': 950, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19053, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3060}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:20.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:04:20.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'18775", 'reported_seq': 18860, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:14.094929+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:14.094929+0000', 'last_peered': '2024-09-18T12:04:14.094929+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:14.094929+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'8700", 'ondisk_log_start': "39'8700", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1733842, 'num_objects': 425, 'num_object_clones': 0, 'num_object_copies': 850, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18773, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:20.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:04:20.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19081", 'reported_seq': 19170, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:14.139063+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:14.139063+0000', 'last_peered': '2024-09-18T12:04:14.139063+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:14.139063+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9000", 'ondisk_log_start': "39'9000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1767844, 'num_objects': 435, 'num_object_clones': 0, 'num_object_copies': 870, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 435, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19078, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3088}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:20.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:04:21.819 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:21.819 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29625/13012 objects degraded (227.674%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":661,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait","count":1}],"num_pgs":24,"num_pools":2,"num_objects":6506,"data_bytes":26566751,"bytes_used":1183346688,"bytes_avail":288726945792,"bytes_total":289910292480,"degraded_objects":29625,"degraded_total":13012,"degraded_ratio":2.2767445434983093,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":426},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:04:21.836 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:04:21.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:22.427 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:22.438 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:22.447 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-18T12:04:21.258123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21651551,"num_objects":5306,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":289913,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13058,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160621,"ondisk_log_size":160621,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1140592,"kb_used_data":217940,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":281974928,"statfs":{"total":289910292480,"available":288742326272,"internally_reserved":0,"allocated":223170560,"data_stored":222376256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,8,3,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22958080,"num_objects":-5605,"num_object_clones":0,"num_object_copies":-11210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003973"},"pg_stats":[{"pgid":"2.f","version":"43'19226","reported_seq":19301,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:19.177040+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:19.177040+0000","last_peered":"2024-09-18T12:04:19.177040+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:19.177040+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1269760,"num_objects":310,"num_object_clones":0,"num_object_copies":620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19226,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3289}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19120","reported_seq":19282,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:18.925195+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:18.925195+0000","last_peered":"2024-09-18T12:04:18.925195+0000","last_clean":"2024-09-18T12:04:18.925195+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:18.925195+0000","last_undegraded":"2024-09-18T12:04:18.925195+0000","last_fullsized":"2024-09-18T12:04:18.925195+0000","mapping_epoch":29,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1277952,"num_objects":312,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16312,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19224","reported_seq":19412,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:18.997427+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:18.997427+0000","last_peered":"2024-09-18T12:04:18.997427+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:18.997427+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363992,"num_objects":334,"num_object_clones":0,"num_object_copies":668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16082,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'18927","reported_seq":18960,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:18.956078+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:18.956078+0000","last_peered":"2024-09-18T12:04:18.956078+0000","last_clean":"2024-09-18T12:04:18.956078+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:18.956078+0000","last_undegraded":"2024-09-18T12:04:18.956078+0000","last_fullsized":"2024-09-18T12:04:18.956078+0000","mapping_epoch":29,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1363968,"num_objects":333,"num_object_clones":0,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18927,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19147","reported_seq":19342,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:19.239791+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:19.239791+0000","last_peered":"2024-09-18T12:04:19.239791+0000","last_clean":"2024-09-18T12:04:19.239791+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:19.239791+0000","last_undegraded":"2024-09-18T12:04:19.239791+0000","last_fullsized":"2024-09-18T12:04:19.239791+0000","mapping_epoch":29,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1228800,"num_objects":300,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16312,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19094","reported_seq":19179,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:19.239172+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:19.239172+0000","last_peered":"2024-09-18T12:04:19.239172+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:19.239172+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1348818,"num_objects":331,"num_object_clones":0,"num_object_copies":662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19092,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3196}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19060","reported_seq":19151,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:19.301456+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:19.301456+0000","last_peered":"2024-09-18T12:04:19.301456+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:19.301456+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1279186,"num_objects":314,"num_object_clones":0,"num_object_copies":628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19058,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19315","reported_seq":19544,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:19.241931+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:19.241931+0000","last_peered":"2024-09-18T12:04:19.241931+0000","last_clean":"2024-09-18T12:04:19.241931+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:19.241931+0000","last_undegraded":"2024-09-18T12:04:19.241931+0000","last_fullsized":"2024-09-18T12:04:19.241931+0000","mapping_epoch":29,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1175552,"num_objects":287,"num_object_clones":0,"num_object_copies":574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15632,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'18985","reported_seq":26115,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:16.375774+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:16.375774+0000","last_peered":"2024-09-18T12:04:16.375774+0000","last_clean":"2024-09-18T12:04:16.375774+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:16.375774+0000","last_undegraded":"2024-09-18T12:04:16.375774+0000","last_fullsized":"2024-09-18T12:04:16.375774+0000","mapping_epoch":38,"log_start":"39'8900","ondisk_log_start":"39'8900","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1721554,"num_objects":422,"num_object_clones":0,"num_object_copies":844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18983,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19129","reported_seq":24078,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:19.294694+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:19.294694+0000","last_peered":"2024-09-18T12:04:19.294694+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:19.294694+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1299666,"num_objects":319,"num_object_clones":0,"num_object_copies":638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19127,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2435,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-782}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19117","reported_seq":19177,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:18.863178+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:18.863178+0000","last_peered":"2024-09-18T12:04:18.863178+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:18.863178+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1418450,"num_objects":348,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19115,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3202}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19162","reported_seq":26575,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:18.973653+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:18.973653+0000","last_peered":"2024-09-18T12:04:18.973653+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:18.973653+0000","last_undegraded":"2024-09-18T12:04:18.973653+0000","last_fullsized":"2024-09-18T12:04:18.973653+0000","mapping_epoch":42,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":336,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15642,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":336}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19204","reported_seq":19290,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:19.207414+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:19.207414+0000","last_peered":"2024-09-18T12:04:19.207414+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:19.207414+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1327721,"num_objects":325,"num_object_clones":0,"num_object_copies":650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19203,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19111","reported_seq":26261,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:16.323703+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:16.323703+0000","last_peered":"2024-09-18T12:04:16.323703+0000","last_clean":"2024-09-18T12:04:16.323703+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:16.323703+0000","last_undegraded":"2024-09-18T12:04:16.323703+0000","last_fullsized":"2024-09-18T12:04:16.323703+0000","mapping_epoch":40,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1685307,"num_objects":414,"num_object_clones":0,"num_object_copies":828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19108,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'18894","reported_seq":18979,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:19.264504+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:19.264504+0000","last_peered":"2024-09-18T12:04:19.264504+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:19.264504+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'8800","ondisk_log_start":"39'8800","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1246418,"num_objects":306,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18892,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19203","reported_seq":19290,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:19.254900+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:19.254900+0000","last_peered":"2024-09-18T12:04:19.254900+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:19.254900+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1268132,"num_objects":313,"num_object_clones":0,"num_object_copies":626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19200,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":21651532,"num_objects":5304,"num_object_clones":0,"num_object_copies":10608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5304,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":289911,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13057,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":220049408,"data_stored":219985600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160618,"ondisk_log_size":160618,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051709,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439376,"kb_used_data":124872,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93932464,"statfs":{"total":96636764160,"available":96186843136,"internally_reserved":0,"allocated":127868928,"data_stored":127620293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.33000000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.27300000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.41699999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855471,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330368,"kb_used_data":20024,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94041472,"statfs":{"total":96636764160,"available":96298467328,"internally_reserved":0,"allocated":20504576,"data_stored":20207092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.216}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.316},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.37}]}]},{"osd":0,"up_from":23,"seq":98784247936,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370848,"kb_used_data":73044,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94000992,"statfs":{"total":96636764160,"available":96257015808,"internally_reserved":0,"allocated":74797056,"data_stored":74548871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304934406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.84199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73273344,"data_stored":73269865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19869696,"data_stored":19816918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126906368,"data_stored":126898817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:25.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:26.103 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:26.107 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:26.132 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-18T12:04:25.259030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12607583,"num_objects":3098,"num_object_clones":0,"num_object_copies":6196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292121,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13125,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160729,"ondisk_log_size":160729,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1106384,"kb_used_data":200368,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":282009136,"statfs":{"total":289910292480,"available":288777355264,"internally_reserved":0,"allocated":205176832,"data_stored":204382528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,4,5,12,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25767936,"num_objects":-6291,"num_object_clones":0,"num_object_copies":-12582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003898"},"pg_stats":[{"pgid":"2.f","version":"43'19349","reported_seq":19424,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.244061+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:24.244061+0000","last_peered":"2024-09-18T12:04:24.244061+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:24.244061+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":765952,"num_objects":187,"num_object_clones":0,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19349,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3412}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19245","reported_seq":19407,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:24.010455+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:24.010455+0000","last_peered":"2024-09-18T12:04:24.010455+0000","last_clean":"2024-09-18T12:04:24.010455+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:24.010455+0000","last_undegraded":"2024-09-18T12:04:24.010455+0000","last_fullsized":"2024-09-18T12:04:24.010455+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":765952,"num_objects":187,"num_object_clones":0,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16437,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19355","reported_seq":19545,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.015302+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:24.015302+0000","last_peered":"2024-09-18T12:04:24.015302+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:24.015302+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":827416,"num_objects":203,"num_object_clones":0,"num_object_copies":406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16213,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19063","reported_seq":19096,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:24.032067+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:24.032067+0000","last_peered":"2024-09-18T12:04:24.032067+0000","last_clean":"2024-09-18T12:04:24.032067+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:24.032067+0000","last_undegraded":"2024-09-18T12:04:24.032067+0000","last_fullsized":"2024-09-18T12:04:24.032067+0000","mapping_epoch":29,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":806912,"num_objects":197,"num_object_clones":0,"num_object_copies":394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19063,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19277","reported_seq":19472,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:24.268823+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:24.268823+0000","last_peered":"2024-09-18T12:04:24.268823+0000","last_clean":"2024-09-18T12:04:24.268823+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:24.268823+0000","last_undegraded":"2024-09-18T12:04:24.268823+0000","last_fullsized":"2024-09-18T12:04:24.268823+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":696320,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16442,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19248","reported_seq":19333,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.209351+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:24.209351+0000","last_peered":"2024-09-18T12:04:24.209351+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:24.209351+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":718034,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19246,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19216","reported_seq":19295,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.294845+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:24.294845+0000","last_peered":"2024-09-18T12:04:24.294845+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:24.294845+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":640210,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19214,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19431","reported_seq":19660,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:24.300747+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:24.300747+0000","last_peered":"2024-09-18T12:04:24.300747+0000","last_clean":"2024-09-18T12:04:24.300747+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:24.300747+0000","last_undegraded":"2024-09-18T12:04:24.300747+0000","last_fullsized":"2024-09-18T12:04:24.300747+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":700416,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15748,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19135","reported_seq":26266,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:21.426667+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:21.426667+0000","last_peered":"2024-09-18T12:04:21.426667+0000","last_clean":"2024-09-18T12:04:21.426667+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:21.426667+0000","last_undegraded":"2024-09-18T12:04:21.426667+0000","last_fullsized":"2024-09-18T12:04:21.426667+0000","mapping_epoch":38,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107154,"num_objects":272,"num_object_clones":0,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19133,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19286","reported_seq":24369,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.315601+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:24.315601+0000","last_peered":"2024-09-18T12:04:24.315601+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:24.315601+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656594,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19284,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2502,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-872}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19256","reported_seq":19311,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:23.977363+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:23.977363+0000","last_peered":"2024-09-18T12:04:23.977363+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:23.977363+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":849106,"num_objects":209,"num_object_clones":0,"num_object_copies":418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19254,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3341}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19304","reported_seq":26717,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:23.975238+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:23.975238+0000","last_peered":"2024-09-18T12:04:23.975238+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:23.975238+0000","last_undegraded":"2024-09-18T12:04:23.975238+0000","last_fullsized":"2024-09-18T12:04:23.975238+0000","mapping_epoch":42,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":794624,"num_objects":194,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15784,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19361","reported_seq":19447,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.229041+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:24.229041+0000","last_peered":"2024-09-18T12:04:24.229041+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:24.229041+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":684649,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19360,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19244","reported_seq":26394,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:21.417022+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:21.417022+0000","last_peered":"2024-09-18T12:04:21.417022+0000","last_clean":"2024-09-18T12:04:21.417022+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:21.417022+0000","last_undegraded":"2024-09-18T12:04:21.417022+0000","last_fullsized":"2024-09-18T12:04:21.417022+0000","mapping_epoch":40,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1140539,"num_objects":281,"num_object_clones":0,"num_object_copies":562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19241,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19020","reported_seq":19105,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.158343+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:24.158343+0000","last_peered":"2024-09-18T12:04:24.158343+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:24.158343+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":730322,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19018,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3413}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19336","reported_seq":19425,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.244282+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:24.244282+0000","last_peered":"2024-09-18T12:04:24.244282+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:24.244282+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":723364,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19333,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3343}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12607564,"num_objects":3096,"num_object_clones":0,"num_object_copies":6192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292119,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13124,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202022912,"data_stored":201959104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160726,"ondisk_log_size":160726,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051710,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422632,"kb_used_data":120608,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93949208,"statfs":{"total":96636764160,"available":96203988992,"internally_reserved":0,"allocated":123502592,"data_stored":123253957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":5.157},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":5.0780000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":5.1820000000000004},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":5.415}]}]},{"osd":1,"up_from":35,"seq":150323855472,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315264,"kb_used_data":13240,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94056576,"statfs":{"total":96636764160,"available":96313933824,"internally_reserved":0,"allocated":13557760,"data_stored":13260276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.38600000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.35999999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368488,"kb_used_data":66520,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":94003352,"statfs":{"total":96636764160,"available":96259432448,"internally_reserved":0,"allocated":68116480,"data_stored":67868295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":3.649},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.432},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":5.2629999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66576384,"data_stored":66572905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12873728,"data_stored":12820950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":122572800,"data_stored":122565249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:26.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:26.861 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:26.882 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:26.890 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-18T12:04:25.259030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12607583,"num_objects":3098,"num_object_clones":0,"num_object_copies":6196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292121,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13125,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160729,"ondisk_log_size":160729,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1106384,"kb_used_data":200368,"kb_used_omap":13,"kb_used_meta":905970,"kb_avail":282009136,"statfs":{"total":289910292480,"available":288777355264,"internally_reserved":0,"allocated":205176832,"data_stored":204382528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":927713810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,4,5,12,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25767936,"num_objects":-6291,"num_object_clones":0,"num_object_copies":-12582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6291,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003898"},"pg_stats":[{"pgid":"2.f","version":"43'19349","reported_seq":19424,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.244061+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:24.244061+0000","last_peered":"2024-09-18T12:04:24.244061+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:24.244061+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":765952,"num_objects":187,"num_object_clones":0,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19349,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3412}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19245","reported_seq":19407,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:24.010455+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:24.010455+0000","last_peered":"2024-09-18T12:04:24.010455+0000","last_clean":"2024-09-18T12:04:24.010455+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:24.010455+0000","last_undegraded":"2024-09-18T12:04:24.010455+0000","last_fullsized":"2024-09-18T12:04:24.010455+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":765952,"num_objects":187,"num_object_clones":0,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16437,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19355","reported_seq":19545,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.015302+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:24.015302+0000","last_peered":"2024-09-18T12:04:24.015302+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:24.015302+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":827416,"num_objects":203,"num_object_clones":0,"num_object_copies":406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16213,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19063","reported_seq":19096,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:24.032067+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:24.032067+0000","last_peered":"2024-09-18T12:04:24.032067+0000","last_clean":"2024-09-18T12:04:24.032067+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:24.032067+0000","last_undegraded":"2024-09-18T12:04:24.032067+0000","last_fullsized":"2024-09-18T12:04:24.032067+0000","mapping_epoch":29,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":806912,"num_objects":197,"num_object_clones":0,"num_object_copies":394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19063,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19277","reported_seq":19472,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:24.268823+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:24.268823+0000","last_peered":"2024-09-18T12:04:24.268823+0000","last_clean":"2024-09-18T12:04:24.268823+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:24.268823+0000","last_undegraded":"2024-09-18T12:04:24.268823+0000","last_fullsized":"2024-09-18T12:04:24.268823+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":696320,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16442,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19248","reported_seq":19333,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.209351+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:24.209351+0000","last_peered":"2024-09-18T12:04:24.209351+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:24.209351+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":718034,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19246,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19216","reported_seq":19295,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.294845+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:24.294845+0000","last_peered":"2024-09-18T12:04:24.294845+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:24.294845+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":640210,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19214,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19431","reported_seq":19660,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:24.300747+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:24.300747+0000","last_peered":"2024-09-18T12:04:24.300747+0000","last_clean":"2024-09-18T12:04:24.300747+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:24.300747+0000","last_undegraded":"2024-09-18T12:04:24.300747+0000","last_fullsized":"2024-09-18T12:04:24.300747+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":700416,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15748,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19135","reported_seq":26266,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:21.426667+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:21.426667+0000","last_peered":"2024-09-18T12:04:21.426667+0000","last_clean":"2024-09-18T12:04:21.426667+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:21.426667+0000","last_undegraded":"2024-09-18T12:04:21.426667+0000","last_fullsized":"2024-09-18T12:04:21.426667+0000","mapping_epoch":38,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1107154,"num_objects":272,"num_object_clones":0,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19133,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19286","reported_seq":24369,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.315601+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:24.315601+0000","last_peered":"2024-09-18T12:04:24.315601+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:24.315601+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656594,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19284,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2502,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-872}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19256","reported_seq":19311,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:23.977363+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:23.977363+0000","last_peered":"2024-09-18T12:04:23.977363+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:23.977363+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":849106,"num_objects":209,"num_object_clones":0,"num_object_copies":418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19254,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3341}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19304","reported_seq":26717,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:23.975238+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:23.975238+0000","last_peered":"2024-09-18T12:04:23.975238+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:23.975238+0000","last_undegraded":"2024-09-18T12:04:23.975238+0000","last_fullsized":"2024-09-18T12:04:23.975238+0000","mapping_epoch":42,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":794624,"num_objects":194,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15784,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19361","reported_seq":19447,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.229041+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:24.229041+0000","last_peered":"2024-09-18T12:04:24.229041+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:24.229041+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":684649,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19360,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19244","reported_seq":26394,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:21.417022+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:21.417022+0000","last_peered":"2024-09-18T12:04:21.417022+0000","last_clean":"2024-09-18T12:04:21.417022+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:21.417022+0000","last_undegraded":"2024-09-18T12:04:21.417022+0000","last_fullsized":"2024-09-18T12:04:21.417022+0000","mapping_epoch":40,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1140539,"num_objects":281,"num_object_clones":0,"num_object_copies":562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19241,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19020","reported_seq":19105,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.158343+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:24.158343+0000","last_peered":"2024-09-18T12:04:24.158343+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:24.158343+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":730322,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19018,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3413}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19336","reported_seq":19425,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.244282+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:24.244282+0000","last_peered":"2024-09-18T12:04:24.244282+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:24.244282+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":723364,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19333,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3343}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12607564,"num_objects":3096,"num_object_clones":0,"num_object_copies":6192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292119,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13124,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202022912,"data_stored":201959104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160726,"ondisk_log_size":160726,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051710,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422632,"kb_used_data":120608,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93949208,"statfs":{"total":96636764160,"available":96203988992,"internally_reserved":0,"allocated":123502592,"data_stored":123253957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":5.157},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":5.0780000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":5.1820000000000004},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":5.415}]}]},{"osd":1,"up_from":35,"seq":150323855472,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315264,"kb_used_data":13240,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94056576,"statfs":{"total":96636764160,"available":96313933824,"internally_reserved":0,"allocated":13557760,"data_stored":13260276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,8,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.5,"15min":0.46899999999999997},"min":{"1min":0.19900000000000001,"5min":0.186,"15min":0.17399999999999999},"max":{"1min":8.6210000000000004,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.665,"5min":1.671,"15min":1.272},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.5190000000000001,"5min":12.244999999999999,"15min":12.244999999999999},"last":0.38600000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:03:21 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.893,"15min":1.3129999999999999},"min":{"1min":0.23699999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.4160000000000004,"5min":13.737,"15min":13.737},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.78500000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.20999999999999999},"max":{"1min":6.7060000000000004,"5min":8.4730000000000008,"15min":11.996},"last":0.35999999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368488,"kb_used_data":66520,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":94003352,"statfs":{"total":96636764160,"available":96259432448,"internally_reserved":0,"allocated":68116480,"data_stored":67868295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":3.649},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.432},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":5.2629999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66576384,"data_stored":66572905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12873728,"data_stored":12820950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":122572800,"data_stored":122565249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:26.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:04:26.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19349", 'reported_seq': 19424, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:24.244061+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:24.244061+0000', 'last_peered': '2024-09-18T12:04:24.244061+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:24.244061+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 765952, 'num_objects': 187, 'num_object_clones': 0, 'num_object_copies': 374, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 187, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19349, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3412}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:26.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:04:26.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19355", 'reported_seq': 19545, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:24.015302+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:24.015302+0000', 'last_peered': '2024-09-18T12:04:24.015302+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:24.015302+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 827416, 'num_objects': 203, 'num_object_clones': 0, 'num_object_copies': 406, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 203, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16213, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3333}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:26.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:04:26.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19248", 'reported_seq': 19333, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:24.209351+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:24.209351+0000', 'last_peered': '2024-09-18T12:04:24.209351+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:24.209351+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9200", 'ondisk_log_start': "39'9200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 718034, 'num_objects': 177, 'num_object_clones': 0, 'num_object_copies': 354, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19246, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3350}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:26.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:04:26.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19216", 'reported_seq': 19295, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:24.294845+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:24.294845+0000', 'last_peered': '2024-09-18T12:04:24.294845+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:24.294845+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9200", 'ondisk_log_start': "39'9200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 640210, 'num_objects': 158, 'num_object_clones': 0, 'num_object_copies': 316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19214, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3401}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:26.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:04:26.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'19286", 'reported_seq': 24369, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:24.315601+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:04:24.315601+0000', 'last_peered': '2024-09-18T12:04:24.315601+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:04:24.315601+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'9200", 'ondisk_log_start': "39'9200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 656594, 'num_objects': 162, 'num_object_clones': 0, 'num_object_copies': 324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1034, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 162, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19284, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2502, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -872}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:26.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:04:26.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19256", 'reported_seq': 19311, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:23.977363+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:04:23.977363+0000', 'last_peered': '2024-09-18T12:04:23.977363+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:04:23.977363+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9200", 'ondisk_log_start': "39'9200", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 849106, 'num_objects': 209, 'num_object_clones': 0, 'num_object_copies': 418, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 209, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19254, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3341}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:26.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:04:26.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19304", 'reported_seq': 26717, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:23.975238+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:23.975238+0000', 'last_peered': '2024-09-18T12:04:23.975238+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:23.975238+0000', 'last_undegraded': '2024-09-18T12:04:23.975238+0000', 'last_fullsized': '2024-09-18T12:04:23.975238+0000', 'mapping_epoch': 42, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 794624, 'num_objects': 194, 'num_object_clones': 0, 'num_object_copies': 388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15784, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 194}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:26.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:04:26.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19361", 'reported_seq': 19447, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:24.229041+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:24.229041+0000', 'last_peered': '2024-09-18T12:04:24.229041+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:24.229041+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 684649, 'num_objects': 168, 'num_object_clones': 0, 'num_object_copies': 336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19360, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3367}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:26.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:04:26.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19020", 'reported_seq': 19105, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:24.158343+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:24.158343+0000', 'last_peered': '2024-09-18T12:04:24.158343+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:24.158343+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9000", 'ondisk_log_start': "39'9000", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 730322, 'num_objects': 180, 'num_object_clones': 0, 'num_object_copies': 360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19018, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3413}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:26.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:04:26.896 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19336", 'reported_seq': 19425, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:24.244282+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:24.244282+0000', 'last_peered': '2024-09-18T12:04:24.244282+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:24.244282+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 723364, 'num_objects': 180, 'num_object_clones': 0, 'num_object_copies': 360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19333, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3343}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:26.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:04:27.619 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:27.619 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29469/6196 objects degraded (475.613%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":667,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":3098,"data_bytes":12607583,"bytes_used":1132937216,"bytes_avail":288777355264,"bytes_total":289910292480,"degraded_objects":29469,"degraded_total":6196,"degraded_ratio":4.7561329890251773,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":241,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":524},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:04:27.656 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:04:27.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:28.414 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:28.421 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:28.447 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-18T12:04:27.259487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11456607,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292402,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13125,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1103920,"kb_used_data":193744,"kb_used_omap":13,"kb_used_meta":910130,"kb_avail":282011600,"statfs":{"total":289910292480,"available":288779878400,"internally_reserved":0,"allocated":198393856,"data_stored":197599552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":931973650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,4,7,10,9,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19132416,"num_objects":-4671,"num_object_clones":0,"num_object_copies":-9342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4671,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003922"},"pg_stats":[{"pgid":"2.f","version":"43'19349","reported_seq":19424,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.244061+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:24.244061+0000","last_peered":"2024-09-18T12:04:24.244061+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:24.244061+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":765952,"num_objects":187,"num_object_clones":0,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19349,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3412}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19245","reported_seq":19407,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:24.010455+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:24.010455+0000","last_peered":"2024-09-18T12:04:24.010455+0000","last_clean":"2024-09-18T12:04:24.010455+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:24.010455+0000","last_undegraded":"2024-09-18T12:04:24.010455+0000","last_fullsized":"2024-09-18T12:04:24.010455+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":765952,"num_objects":187,"num_object_clones":0,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16437,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19355","reported_seq":19545,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.015302+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:24.015302+0000","last_peered":"2024-09-18T12:04:24.015302+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:24.015302+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":827416,"num_objects":203,"num_object_clones":0,"num_object_copies":406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16213,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19063","reported_seq":19096,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:24.032067+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:24.032067+0000","last_peered":"2024-09-18T12:04:24.032067+0000","last_clean":"2024-09-18T12:04:24.032067+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:24.032067+0000","last_undegraded":"2024-09-18T12:04:24.032067+0000","last_fullsized":"2024-09-18T12:04:24.032067+0000","mapping_epoch":29,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":806912,"num_objects":197,"num_object_clones":0,"num_object_copies":394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19063,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19277","reported_seq":19472,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:24.268823+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:24.268823+0000","last_peered":"2024-09-18T12:04:24.268823+0000","last_clean":"2024-09-18T12:04:24.268823+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:24.268823+0000","last_undegraded":"2024-09-18T12:04:24.268823+0000","last_fullsized":"2024-09-18T12:04:24.268823+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":696320,"num_objects":170,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16442,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19248","reported_seq":19333,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.209351+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:24.209351+0000","last_peered":"2024-09-18T12:04:24.209351+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:24.209351+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":718034,"num_objects":177,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19246,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19216","reported_seq":19295,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.294845+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:24.294845+0000","last_peered":"2024-09-18T12:04:24.294845+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:24.294845+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":640210,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19214,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19431","reported_seq":19660,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:24.300747+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:24.300747+0000","last_peered":"2024-09-18T12:04:24.300747+0000","last_clean":"2024-09-18T12:04:24.300747+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:24.300747+0000","last_undegraded":"2024-09-18T12:04:24.300747+0000","last_fullsized":"2024-09-18T12:04:24.300747+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":700416,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15748,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19270","reported_seq":26402,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:26.471881+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:26.471881+0000","last_peered":"2024-09-18T12:04:26.471881+0000","last_clean":"2024-09-18T12:04:26.471881+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:26.471881+0000","last_undegraded":"2024-09-18T12:04:26.471881+0000","last_fullsized":"2024-09-18T12:04:26.471881+0000","mapping_epoch":38,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554194,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19268,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19286","reported_seq":24369,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.315601+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:24.315601+0000","last_peered":"2024-09-18T12:04:24.315601+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:24.315601+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656594,"num_objects":162,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19284,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2502,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-872}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19256","reported_seq":19311,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:23.977363+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:23.977363+0000","last_peered":"2024-09-18T12:04:23.977363+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:23.977363+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":849106,"num_objects":209,"num_object_clones":0,"num_object_copies":418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19254,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3341}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19304","reported_seq":26717,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:23.975238+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:23.975238+0000","last_peered":"2024-09-18T12:04:23.975238+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:23.975238+0000","last_undegraded":"2024-09-18T12:04:23.975238+0000","last_fullsized":"2024-09-18T12:04:23.975238+0000","mapping_epoch":42,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":794624,"num_objects":194,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15784,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19361","reported_seq":19447,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.229041+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:24.229041+0000","last_peered":"2024-09-18T12:04:24.229041+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:24.229041+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":684649,"num_objects":168,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19360,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19390","reported_seq":26540,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:26.505585+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:26.505585+0000","last_peered":"2024-09-18T12:04:26.505585+0000","last_clean":"2024-09-18T12:04:26.505585+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:26.505585+0000","last_undegraded":"2024-09-18T12:04:26.505585+0000","last_fullsized":"2024-09-18T12:04:26.505585+0000","mapping_epoch":40,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":542523,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19387,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19020","reported_seq":19105,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.158343+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:24.158343+0000","last_peered":"2024-09-18T12:04:24.158343+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:24.158343+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9000","ondisk_log_start":"39'9000","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":730322,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19018,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3413}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19336","reported_seq":19425,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:24.244282+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:24.244282+0000","last_peered":"2024-09-18T12:04:24.244282+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:24.244282+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":723364,"num_objects":180,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19333,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3343}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":11456588,"num_objects":2815,"num_object_clones":0,"num_object_copies":5630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292400,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13124,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":195096576,"data_stored":195032768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160807,"ondisk_log_size":160807,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051710,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422632,"kb_used_data":120608,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93949208,"statfs":{"total":96636764160,"available":96203988992,"internally_reserved":0,"allocated":123502592,"data_stored":123253957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":5.157},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":5.0780000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":5.1820000000000004},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":5.415}]}]},{"osd":1,"up_from":35,"seq":150323855473,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312800,"kb_used_data":6616,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94059040,"statfs":{"total":96636764160,"available":96316456960,"internally_reserved":0,"allocated":6774784,"data_stored":6477300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.311},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.34200000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368488,"kb_used_data":66520,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":94003352,"statfs":{"total":96636764160,"available":96259432448,"internally_reserved":0,"allocated":68116480,"data_stored":67868295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309194246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:03:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.268,"15min":1.0780000000000001},"min":{"1min":0.21099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":4.5259999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":3.649},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":1.046},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":5.875,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.20899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.432},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":5.2629999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66576384,"data_stored":66572905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5947392,"data_stored":5894614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":122572800,"data_stored":122565249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:30.909 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Removed 155559 objects 2024-09-18T12:04:30.909 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi154.stdout:Clean up completed and total clean up time :364.731 2024-09-18T12:04:31.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:31.752 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:31.755 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:31.765 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-18T12:04:31.264349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3776607,"num_objects":942,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":294277,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13237,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1104480,"kb_used_data":181820,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282011040,"statfs":{"total":289910292480,"available":288779304960,"internally_reserved":0,"allocated":186183680,"data_stored":185389376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,8,4,9,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22790144,"num_objects":-5564,"num_object_clones":0,"num_object_copies":-11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5564,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006686"},"pg_stats":[{"pgid":"2.f","version":"43'19504","reported_seq":19579,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.305761+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:29.305761+0000","last_peered":"2024-09-18T12:04:29.305761+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:29.305761+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19504,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3567}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19381","reported_seq":19543,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:28.935964+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:28.935964+0000","last_peered":"2024-09-18T12:04:28.935964+0000","last_clean":"2024-09-18T12:04:28.935964+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:28.935964+0000","last_undegraded":"2024-09-18T12:04:28.935964+0000","last_fullsized":"2024-09-18T12:04:28.935964+0000","mapping_epoch":29,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208896,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16573,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19506","reported_seq":19695,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.018519+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:29.018519+0000","last_peered":"2024-09-18T12:04:29.018519+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:29.018519+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208920,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16364,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3484}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19196","reported_seq":19229,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:29.027787+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:29.027787+0000","last_peered":"2024-09-18T12:04:29.027787+0000","last_clean":"2024-09-18T12:04:29.027787+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:29.027787+0000","last_undegraded":"2024-09-18T12:04:29.027787+0000","last_fullsized":"2024-09-18T12:04:29.027787+0000","mapping_epoch":29,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19419","reported_seq":19613,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:29.362702+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:29.362702+0000","last_peered":"2024-09-18T12:04:29.362702+0000","last_clean":"2024-09-18T12:04:29.362702+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:29.362702+0000","last_undegraded":"2024-09-18T12:04:29.362702+0000","last_fullsized":"2024-09-18T12:04:29.362702+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16584,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19373","reported_seq":19458,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.236699+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:29.236699+0000","last_peered":"2024-09-18T12:04:29.236699+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:29.236699+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206034,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19371,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3475}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19333","reported_seq":19424,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.357319+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:29.357319+0000","last_peered":"2024-09-18T12:04:29.357319+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:29.357319+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160978,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19331,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19563","reported_seq":19792,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:29.322497+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:29.322497+0000","last_peered":"2024-09-18T12:04:29.322497+0000","last_clean":"2024-09-18T12:04:29.322497+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:29.322497+0000","last_undegraded":"2024-09-18T12:04:29.322497+0000","last_fullsized":"2024-09-18T12:04:29.322497+0000","mapping_epoch":29,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159744,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19270","reported_seq":26402,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:26.471881+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:26.471881+0000","last_peered":"2024-09-18T12:04:26.471881+0000","last_clean":"2024-09-18T12:04:26.471881+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:26.471881+0000","last_undegraded":"2024-09-18T12:04:26.471881+0000","last_fullsized":"2024-09-18T12:04:26.471881+0000","mapping_epoch":38,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554194,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19268,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19407","reported_seq":24714,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.295123+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:29.295123+0000","last_peered":"2024-09-18T12:04:29.295123+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:29.295123+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160978,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19405,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-881}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19398","reported_seq":19458,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:28.990503+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:28.990503+0000","last_peered":"2024-09-18T12:04:28.990503+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:28.990503+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267474,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19396,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19447","reported_seq":26860,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:28.960521+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:28.960521+0000","last_peered":"2024-09-18T12:04:28.960521+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:28.960521+0000","last_undegraded":"2024-09-18T12:04:28.960521+0000","last_fullsized":"2024-09-18T12:04:28.960521+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208896,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15927,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":51}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19482","reported_seq":19568,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.327797+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:29.327797+0000","last_peered":"2024-09-18T12:04:29.327797+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:29.327797+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189033,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19481,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3488}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19390","reported_seq":26540,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:26.505585+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:26.505585+0000","last_peered":"2024-09-18T12:04:26.505585+0000","last_clean":"2024-09-18T12:04:26.505585+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:26.505585+0000","last_undegraded":"2024-09-18T12:04:26.505585+0000","last_fullsized":"2024-09-18T12:04:26.505585+0000","mapping_epoch":40,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":542523,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19387,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19152","reported_seq":19237,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.236836+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:29.236836+0000","last_peered":"2024-09-18T12:04:29.236836+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:29.236836+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189650,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19150,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19461","reported_seq":19550,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.257992+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:29.257992+0000","last_peered":"2024-09-18T12:04:29.257992+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:29.257992+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211364,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19458,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3776588,"num_objects":940,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":294275,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13236,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184078336,"data_stored":184014528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051711,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422360,"kb_used_data":116176,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93949480,"statfs":{"total":96636764160,"available":96204267520,"internally_reserved":0,"allocated":118964224,"data_stored":118715589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.26700000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.191}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.23799999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855473,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312800,"kb_used_data":6616,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94059040,"statfs":{"total":96636764160,"available":96316456960,"internally_reserved":0,"allocated":6774784,"data_stored":6477300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.311},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.34200000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369320,"kb_used_data":59028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94002520,"statfs":{"total":96636764160,"available":96258580480,"internally_reserved":0,"allocated":60444672,"data_stored":60196487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.38400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":60051456,"data_stored":60047977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5947392,"data_stored":5894614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":118079488,"data_stored":118071937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:31.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:32.066 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:32.069 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:32.079 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-18T12:04:31.264349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3776607,"num_objects":942,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":294277,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13237,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1104480,"kb_used_data":181820,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282011040,"statfs":{"total":289910292480,"available":288779304960,"internally_reserved":0,"allocated":186183680,"data_stored":185389376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,8,4,9,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22790144,"num_objects":-5564,"num_object_clones":0,"num_object_copies":-11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5564,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006686"},"pg_stats":[{"pgid":"2.f","version":"43'19504","reported_seq":19579,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.305761+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:29.305761+0000","last_peered":"2024-09-18T12:04:29.305761+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:29.305761+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19504,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3567}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19381","reported_seq":19543,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:28.935964+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:28.935964+0000","last_peered":"2024-09-18T12:04:28.935964+0000","last_clean":"2024-09-18T12:04:28.935964+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:28.935964+0000","last_undegraded":"2024-09-18T12:04:28.935964+0000","last_fullsized":"2024-09-18T12:04:28.935964+0000","mapping_epoch":29,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208896,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16573,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19506","reported_seq":19695,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.018519+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:29.018519+0000","last_peered":"2024-09-18T12:04:29.018519+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:29.018519+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208920,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16364,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3484}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19196","reported_seq":19229,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:29.027787+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:29.027787+0000","last_peered":"2024-09-18T12:04:29.027787+0000","last_clean":"2024-09-18T12:04:29.027787+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:29.027787+0000","last_undegraded":"2024-09-18T12:04:29.027787+0000","last_fullsized":"2024-09-18T12:04:29.027787+0000","mapping_epoch":29,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19419","reported_seq":19613,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:29.362702+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:29.362702+0000","last_peered":"2024-09-18T12:04:29.362702+0000","last_clean":"2024-09-18T12:04:29.362702+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:29.362702+0000","last_undegraded":"2024-09-18T12:04:29.362702+0000","last_fullsized":"2024-09-18T12:04:29.362702+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16584,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19373","reported_seq":19458,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.236699+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:29.236699+0000","last_peered":"2024-09-18T12:04:29.236699+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:29.236699+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206034,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19371,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3475}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19333","reported_seq":19424,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.357319+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:29.357319+0000","last_peered":"2024-09-18T12:04:29.357319+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:29.357319+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160978,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19331,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19563","reported_seq":19792,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:29.322497+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:29.322497+0000","last_peered":"2024-09-18T12:04:29.322497+0000","last_clean":"2024-09-18T12:04:29.322497+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:29.322497+0000","last_undegraded":"2024-09-18T12:04:29.322497+0000","last_fullsized":"2024-09-18T12:04:29.322497+0000","mapping_epoch":29,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159744,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19270","reported_seq":26402,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:26.471881+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:26.471881+0000","last_peered":"2024-09-18T12:04:26.471881+0000","last_clean":"2024-09-18T12:04:26.471881+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:26.471881+0000","last_undegraded":"2024-09-18T12:04:26.471881+0000","last_fullsized":"2024-09-18T12:04:26.471881+0000","mapping_epoch":38,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554194,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19268,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19407","reported_seq":24714,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.295123+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:29.295123+0000","last_peered":"2024-09-18T12:04:29.295123+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:29.295123+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160978,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19405,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-881}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19398","reported_seq":19458,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:28.990503+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:28.990503+0000","last_peered":"2024-09-18T12:04:28.990503+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:28.990503+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267474,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19396,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19447","reported_seq":26860,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:28.960521+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:28.960521+0000","last_peered":"2024-09-18T12:04:28.960521+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:28.960521+0000","last_undegraded":"2024-09-18T12:04:28.960521+0000","last_fullsized":"2024-09-18T12:04:28.960521+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208896,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15927,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":51}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19482","reported_seq":19568,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.327797+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:29.327797+0000","last_peered":"2024-09-18T12:04:29.327797+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:29.327797+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189033,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19481,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3488}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19390","reported_seq":26540,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:26.505585+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:26.505585+0000","last_peered":"2024-09-18T12:04:26.505585+0000","last_clean":"2024-09-18T12:04:26.505585+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:26.505585+0000","last_undegraded":"2024-09-18T12:04:26.505585+0000","last_fullsized":"2024-09-18T12:04:26.505585+0000","mapping_epoch":40,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":542523,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19387,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19152","reported_seq":19237,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.236836+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:29.236836+0000","last_peered":"2024-09-18T12:04:29.236836+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:29.236836+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189650,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19150,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19461","reported_seq":19550,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.257992+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:29.257992+0000","last_peered":"2024-09-18T12:04:29.257992+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:29.257992+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211364,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19458,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3776588,"num_objects":940,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":294275,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13236,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184078336,"data_stored":184014528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051711,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422360,"kb_used_data":116176,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93949480,"statfs":{"total":96636764160,"available":96204267520,"internally_reserved":0,"allocated":118964224,"data_stored":118715589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.26700000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.191}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.23799999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855473,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312800,"kb_used_data":6616,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94059040,"statfs":{"total":96636764160,"available":96316456960,"internally_reserved":0,"allocated":6774784,"data_stored":6477300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.311},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.34200000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369320,"kb_used_data":59028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94002520,"statfs":{"total":96636764160,"available":96258580480,"internally_reserved":0,"allocated":60444672,"data_stored":60196487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.38400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":60051456,"data_stored":60047977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5947392,"data_stored":5894614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":118079488,"data_stored":118071937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:32.080 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:04:32.081 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19504", 'reported_seq': 19579, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:29.305761+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:29.305761+0000', 'last_peered': '2024-09-18T12:04:29.305761+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:29.305761+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 131072, 'num_objects': 32, 'num_object_clones': 0, 'num_object_copies': 64, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 32, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19504, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3567}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:32.081 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:04:32.081 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19506", 'reported_seq': 19695, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:29.018519+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:29.018519+0000', 'last_peered': '2024-09-18T12:04:29.018519+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:29.018519+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 208920, 'num_objects': 52, 'num_object_clones': 0, 'num_object_copies': 104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 52, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16364, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3484}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:32.081 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:04:32.081 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19373", 'reported_seq': 19458, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:29.236699+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:29.236699+0000', 'last_peered': '2024-09-18T12:04:29.236699+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:29.236699+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 206034, 'num_objects': 52, 'num_object_clones': 0, 'num_object_copies': 104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 52, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19371, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3475}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:32.081 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:04:32.081 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19333", 'reported_seq': 19424, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:29.357319+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:29.357319+0000', 'last_peered': '2024-09-18T12:04:29.357319+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:29.357319+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 160978, 'num_objects': 41, 'num_object_clones': 0, 'num_object_copies': 82, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 41, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19331, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3518}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:32.081 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:04:32.081 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'19407", 'reported_seq': 24714, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:29.295123+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:04:29.295123+0000', 'last_peered': '2024-09-18T12:04:29.295123+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:04:29.295123+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 160978, 'num_objects': 41, 'num_object_clones': 0, 'num_object_copies': 82, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 922, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 41, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19405, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2614, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -881}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:32.082 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:04:32.082 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19398", 'reported_seq': 19458, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:28.990503+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:04:28.990503+0000', 'last_peered': '2024-09-18T12:04:28.990503+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:04:28.990503+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 267474, 'num_objects': 67, 'num_object_clones': 0, 'num_object_copies': 134, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 67, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19396, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3483}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:32.082 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:04:32.082 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19447", 'reported_seq': 26860, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:28.960521+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:28.960521+0000', 'last_peered': '2024-09-18T12:04:28.960521+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:28.960521+0000', 'last_undegraded': '2024-09-18T12:04:28.960521+0000', 'last_fullsized': '2024-09-18T12:04:28.960521+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 208896, 'num_objects': 51, 'num_object_clones': 0, 'num_object_copies': 102, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 51, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15927, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 51}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:32.082 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:04:32.082 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19482", 'reported_seq': 19568, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:29.327797+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:29.327797+0000', 'last_peered': '2024-09-18T12:04:29.327797+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:29.327797+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 189033, 'num_objects': 47, 'num_object_clones': 0, 'num_object_copies': 94, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 47, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19481, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3488}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:32.082 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:04:32.082 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19152", 'reported_seq': 19237, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:29.236836+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:29.236836+0000', 'last_peered': '2024-09-18T12:04:29.236836+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:29.236836+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 189650, 'num_objects': 48, 'num_object_clones': 0, 'num_object_copies': 96, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 48, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19150, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3545}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:32.082 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:04:32.082 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19461", 'reported_seq': 19550, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:29.257992+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:29.257992+0000', 'last_peered': '2024-09-18T12:04:29.257992+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:29.257992+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 211364, 'num_objects': 55, 'num_object_clones': 0, 'num_object_copies': 110, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 55, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19458, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3468}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:32.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:04:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:32.421 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29357/1884 objects degraded (1558.227%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":672,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":942,"data_bytes":3776607,"bytes_used":1130987520,"bytes_avail":288779304960,"bytes_total":289910292480,"degraded_objects":29357,"degraded_total":1884,"degraded_ratio":15.582271762208068,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":268,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":463},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:04:32.435 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:04:32.435 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:32.742 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:32.744 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:32.755 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-18T12:04:31.264349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3776607,"num_objects":942,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":294277,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13237,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160885,"ondisk_log_size":160885,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1104480,"kb_used_data":181820,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282011040,"statfs":{"total":289910292480,"available":288779304960,"internally_reserved":0,"allocated":186183680,"data_stored":185389376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,8,4,9,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22790144,"num_objects":-5564,"num_object_clones":0,"num_object_copies":-11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5564,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006686"},"pg_stats":[{"pgid":"2.f","version":"43'19504","reported_seq":19579,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.305761+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:29.305761+0000","last_peered":"2024-09-18T12:04:29.305761+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:29.305761+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19504,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3567}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19381","reported_seq":19543,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:28.935964+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:28.935964+0000","last_peered":"2024-09-18T12:04:28.935964+0000","last_clean":"2024-09-18T12:04:28.935964+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:28.935964+0000","last_undegraded":"2024-09-18T12:04:28.935964+0000","last_fullsized":"2024-09-18T12:04:28.935964+0000","mapping_epoch":29,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208896,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16573,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19506","reported_seq":19695,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.018519+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:29.018519+0000","last_peered":"2024-09-18T12:04:29.018519+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:29.018519+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208920,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16364,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3484}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19196","reported_seq":19229,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:29.027787+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:29.027787+0000","last_peered":"2024-09-18T12:04:29.027787+0000","last_clean":"2024-09-18T12:04:29.027787+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:29.027787+0000","last_undegraded":"2024-09-18T12:04:29.027787+0000","last_fullsized":"2024-09-18T12:04:29.027787+0000","mapping_epoch":29,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19419","reported_seq":19613,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:29.362702+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:29.362702+0000","last_peered":"2024-09-18T12:04:29.362702+0000","last_clean":"2024-09-18T12:04:29.362702+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:29.362702+0000","last_undegraded":"2024-09-18T12:04:29.362702+0000","last_fullsized":"2024-09-18T12:04:29.362702+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16584,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19373","reported_seq":19458,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.236699+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:29.236699+0000","last_peered":"2024-09-18T12:04:29.236699+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:29.236699+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206034,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19371,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3475}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19333","reported_seq":19424,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.357319+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:29.357319+0000","last_peered":"2024-09-18T12:04:29.357319+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:29.357319+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160978,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19331,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19563","reported_seq":19792,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:29.322497+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:29.322497+0000","last_peered":"2024-09-18T12:04:29.322497+0000","last_clean":"2024-09-18T12:04:29.322497+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:29.322497+0000","last_undegraded":"2024-09-18T12:04:29.322497+0000","last_fullsized":"2024-09-18T12:04:29.322497+0000","mapping_epoch":29,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159744,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15880,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19270","reported_seq":26402,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:26.471881+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:26.471881+0000","last_peered":"2024-09-18T12:04:26.471881+0000","last_clean":"2024-09-18T12:04:26.471881+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:26.471881+0000","last_undegraded":"2024-09-18T12:04:26.471881+0000","last_fullsized":"2024-09-18T12:04:26.471881+0000","mapping_epoch":38,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554194,"num_objects":137,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19268,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19407","reported_seq":24714,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.295123+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:29.295123+0000","last_peered":"2024-09-18T12:04:29.295123+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:29.295123+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160978,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19405,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-881}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19398","reported_seq":19458,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:28.990503+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:28.990503+0000","last_peered":"2024-09-18T12:04:28.990503+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:28.990503+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267474,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19396,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19447","reported_seq":26860,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:28.960521+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:28.960521+0000","last_peered":"2024-09-18T12:04:28.960521+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:28.960521+0000","last_undegraded":"2024-09-18T12:04:28.960521+0000","last_fullsized":"2024-09-18T12:04:28.960521+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208896,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15927,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"1,2","objects":51}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19482","reported_seq":19568,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.327797+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:29.327797+0000","last_peered":"2024-09-18T12:04:29.327797+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:29.327797+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189033,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19481,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3488}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19390","reported_seq":26540,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:26.505585+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:26.505585+0000","last_peered":"2024-09-18T12:04:26.505585+0000","last_clean":"2024-09-18T12:04:26.505585+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:26.505585+0000","last_undegraded":"2024-09-18T12:04:26.505585+0000","last_fullsized":"2024-09-18T12:04:26.505585+0000","mapping_epoch":40,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":542523,"num_objects":135,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19387,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19152","reported_seq":19237,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.236836+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:29.236836+0000","last_peered":"2024-09-18T12:04:29.236836+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:29.236836+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189650,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19150,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19461","reported_seq":19550,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:29.257992+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:29.257992+0000","last_peered":"2024-09-18T12:04:29.257992+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:29.257992+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211364,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19458,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3776588,"num_objects":940,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":294275,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13236,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184078336,"data_stored":184014528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160882,"ondisk_log_size":160882,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051711,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422360,"kb_used_data":116176,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93949480,"statfs":{"total":96636764160,"available":96204267520,"internally_reserved":0,"allocated":118964224,"data_stored":118715589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:03:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.95799999999999996,"15min":0.64300000000000002},"min":{"1min":0.185,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":19.776,"5min":19.776,"15min":19.776},"last":0.26700000000000002},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.8500000000000001,"15min":1.0549999999999999},"min":{"1min":0.217,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":26.702000000000002,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.191}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.23799999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855473,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312800,"kb_used_data":6616,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94059040,"statfs":{"total":96636764160,"available":96316456960,"internally_reserved":0,"allocated":6774784,"data_stored":6477300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.311},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.34200000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":369320,"kb_used_data":59028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94002520,"statfs":{"total":96636764160,"available":96258580480,"internally_reserved":0,"allocated":60444672,"data_stored":60196487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.38400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":60051456,"data_stored":60047977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5947392,"data_stored":5894614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":118079488,"data_stored":118071937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:35.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:36.054 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:36.066 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-09-18T12:04:35.265696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13672,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1102272,"kb_used_data":171304,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":282013248,"statfs":{"total":289910292480,"available":288781565952,"internally_reserved":0,"allocated":175415296,"data_stored":174629136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20480024,"num_objects":-5001,"num_object_clones":0,"num_object_copies":-10002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5001,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007077"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19611,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.866144+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:30.866144+0000","last_peered":"2024-09-18T12:04:30.866144+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:30.866144+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19594,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.851728+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:30.851728+0000","last_peered":"2024-09-18T12:04:30.851728+0000","last_clean":"2024-09-18T12:04:30.851728+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:30.851728+0000","last_undegraded":"2024-09-18T12:04:30.851728+0000","last_fullsized":"2024-09-18T12:04:30.851728+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19748,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.900026+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:30.900026+0000","last_peered":"2024-09-18T12:04:30.900026+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:30.900026+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19293,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.884579+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:30.884579+0000","last_peered":"2024-09-18T12:04:30.884579+0000","last_clean":"2024-09-18T12:04:30.884579+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:30.884579+0000","last_undegraded":"2024-09-18T12:04:30.884579+0000","last_fullsized":"2024-09-18T12:04:30.884579+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19642,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.690185+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:30.690185+0000","last_peered":"2024-09-18T12:04:30.690185+0000","last_clean":"2024-09-18T12:04:30.690185+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:30.690185+0000","last_undegraded":"2024-09-18T12:04:30.690185+0000","last_fullsized":"2024-09-18T12:04:30.690185+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19508,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.850753+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:30.850753+0000","last_peered":"2024-09-18T12:04:30.850753+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:30.850753+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19463,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.794526+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:30.794526+0000","last_peered":"2024-09-18T12:04:30.794526+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:30.794526+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19831,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.833697+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:30.833697+0000","last_peered":"2024-09-18T12:04:30.833697+0000","last_clean":"2024-09-18T12:04:30.833697+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:30.833697+0000","last_undegraded":"2024-09-18T12:04:30.833697+0000","last_fullsized":"2024-09-18T12:04:30.833697+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26537,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.903687+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:30.903687+0000","last_peered":"2024-09-18T12:04:30.903687+0000","last_clean":"2024-09-18T12:04:30.903687+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:30.903687+0000","last_undegraded":"2024-09-18T12:04:30.903687+0000","last_fullsized":"2024-09-18T12:04:30.903687+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":25623,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:34.378336+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:34.378336+0000","last_peered":"2024-09-18T12:04:34.378336+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:34.378336+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3049,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":19523,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.910024+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:30.910024+0000","last_peered":"2024-09-18T12:04:30.910024+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:30.910024+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26911,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:30.872255+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:30.872255+0000","last_peered":"2024-09-18T12:04:30.872255+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:30.872255+0000","last_undegraded":"2024-09-18T12:04:30.872255+0000","last_fullsized":"2024-09-18T12:04:30.872255+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19614,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.878037+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:30.878037+0000","last_peered":"2024-09-18T12:04:30.878037+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:30.878037+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26672,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.816869+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:30.816869+0000","last_peered":"2024-09-18T12:04:30.816869+0000","last_clean":"2024-09-18T12:04:30.816869+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:30.816869+0000","last_undegraded":"2024-09-18T12:04:30.816869+0000","last_fullsized":"2024-09-18T12:04:30.816869+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19283,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.738824+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:30.738824+0000","last_peered":"2024-09-18T12:04:30.738824+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:30.738824+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19601,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.897773+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:30.897773+0000","last_peered":"2024-09-18T12:04:30.897773+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:30.897773+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13671,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174325760,"data_stored":174270096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051712,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420096,"kb_used_data":113912,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951744,"statfs":{"total":96636764160,"available":96206585856,"internally_reserved":0,"allocated":116645888,"data_stored":116401325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.34399999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.52700000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855474,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314916,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":94056924,"statfs":{"total":96636764160,"available":96314290176,"internally_reserved":0,"allocated":421888,"data_stored":128476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.311},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.34200000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367260,"kb_used_data":56980,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004580,"statfs":{"total":96636764160,"available":96260689920,"internally_reserved":0,"allocated":58347520,"data_stored":58099335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":8638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:36.068 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:36.359 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:36.362 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:36.374 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-09-18T12:04:35.265696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13672,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1102272,"kb_used_data":171304,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":282013248,"statfs":{"total":289910292480,"available":288781565952,"internally_reserved":0,"allocated":175415296,"data_stored":174629136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20480024,"num_objects":-5001,"num_object_clones":0,"num_object_copies":-10002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5001,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007077"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19611,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.866144+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:30.866144+0000","last_peered":"2024-09-18T12:04:30.866144+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:30.866144+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19594,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.851728+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:30.851728+0000","last_peered":"2024-09-18T12:04:30.851728+0000","last_clean":"2024-09-18T12:04:30.851728+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:30.851728+0000","last_undegraded":"2024-09-18T12:04:30.851728+0000","last_fullsized":"2024-09-18T12:04:30.851728+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19748,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.900026+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:30.900026+0000","last_peered":"2024-09-18T12:04:30.900026+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:30.900026+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19293,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.884579+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:30.884579+0000","last_peered":"2024-09-18T12:04:30.884579+0000","last_clean":"2024-09-18T12:04:30.884579+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:30.884579+0000","last_undegraded":"2024-09-18T12:04:30.884579+0000","last_fullsized":"2024-09-18T12:04:30.884579+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19642,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.690185+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:30.690185+0000","last_peered":"2024-09-18T12:04:30.690185+0000","last_clean":"2024-09-18T12:04:30.690185+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:30.690185+0000","last_undegraded":"2024-09-18T12:04:30.690185+0000","last_fullsized":"2024-09-18T12:04:30.690185+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19508,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.850753+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:30.850753+0000","last_peered":"2024-09-18T12:04:30.850753+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:30.850753+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19463,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.794526+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:30.794526+0000","last_peered":"2024-09-18T12:04:30.794526+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:30.794526+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19831,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.833697+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:30.833697+0000","last_peered":"2024-09-18T12:04:30.833697+0000","last_clean":"2024-09-18T12:04:30.833697+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:30.833697+0000","last_undegraded":"2024-09-18T12:04:30.833697+0000","last_fullsized":"2024-09-18T12:04:30.833697+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26537,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.903687+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:30.903687+0000","last_peered":"2024-09-18T12:04:30.903687+0000","last_clean":"2024-09-18T12:04:30.903687+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:30.903687+0000","last_undegraded":"2024-09-18T12:04:30.903687+0000","last_fullsized":"2024-09-18T12:04:30.903687+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":25623,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:34.378336+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:34.378336+0000","last_peered":"2024-09-18T12:04:34.378336+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:34.378336+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3049,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":19523,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.910024+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:30.910024+0000","last_peered":"2024-09-18T12:04:30.910024+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:30.910024+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26911,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:30.872255+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:30.872255+0000","last_peered":"2024-09-18T12:04:30.872255+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:30.872255+0000","last_undegraded":"2024-09-18T12:04:30.872255+0000","last_fullsized":"2024-09-18T12:04:30.872255+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19614,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.878037+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:30.878037+0000","last_peered":"2024-09-18T12:04:30.878037+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:30.878037+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26672,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.816869+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:30.816869+0000","last_peered":"2024-09-18T12:04:30.816869+0000","last_clean":"2024-09-18T12:04:30.816869+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:30.816869+0000","last_undegraded":"2024-09-18T12:04:30.816869+0000","last_fullsized":"2024-09-18T12:04:30.816869+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19283,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.738824+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:30.738824+0000","last_peered":"2024-09-18T12:04:30.738824+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:30.738824+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19601,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.897773+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:30.897773+0000","last_peered":"2024-09-18T12:04:30.897773+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:30.897773+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13671,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174325760,"data_stored":174270096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051712,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420096,"kb_used_data":113912,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951744,"statfs":{"total":96636764160,"available":96206585856,"internally_reserved":0,"allocated":116645888,"data_stored":116401325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.34399999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.52700000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855474,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314916,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":94056924,"statfs":{"total":96636764160,"available":96314290176,"internally_reserved":0,"allocated":421888,"data_stored":128476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.311},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.34200000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367260,"kb_used_data":56980,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004580,"statfs":{"total":96636764160,"available":96260689920,"internally_reserved":0,"allocated":58347520,"data_stored":58099335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":8638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:36.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:04:36.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19611, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.866144+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:30.866144+0000', 'last_peered': '2024-09-18T12:04:30.866144+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:30.866144+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:36.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:04:36.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19748, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.900026+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:30.900026+0000', 'last_peered': '2024-09-18T12:04:30.900026+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:30.900026+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:36.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:04:36.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19508, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.850753+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:30.850753+0000', 'last_peered': '2024-09-18T12:04:30.850753+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:30.850753+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:36.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:04:36.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19463, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.794526+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:30.794526+0000', 'last_peered': '2024-09-18T12:04:30.794526+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:30.794526+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:36.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:04:36.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'19446", 'reported_seq': 25623, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:34.378336+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:04:34.378336+0000', 'last_peered': '2024-09-18T12:04:34.378336+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:04:34.378336+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 487, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19444, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3049, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -485}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:36.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:04:36.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19463", 'reported_seq': 19523, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.910024+0000', 'last_change': '2024-09-18T11:55:04.732008+0000', 'last_active': '2024-09-18T12:04:30.910024+0000', 'last_peered': '2024-09-18T12:04:30.910024+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:04:30.910024+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3552, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19461, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3548}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:36.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:04:36.377 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26911, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:30.872255+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:30.872255+0000', 'last_peered': '2024-09-18T12:04:30.872255+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:30.872255+0000', 'last_undegraded': '2024-09-18T12:04:30.872255+0000', 'last_fullsized': '2024-09-18T12:04:30.872255+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:36.377 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:04:36.377 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19614, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.878037+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:30.878037+0000', 'last_peered': '2024-09-18T12:04:30.878037+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:30.878037+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:36.377 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:04:36.377 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 19283, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.738824+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:30.738824+0000', 'last_peered': '2024-09-18T12:04:30.738824+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:30.738824+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:36.377 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:04:36.377 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19601, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.897773+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:30.897773+0000', 'last_peered': '2024-09-18T12:04:30.897773+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:30.897773+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:36.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:04:36.734 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:36.734 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28922/44 objects degraded (65731.818%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":676,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1128726528,"bytes_avail":288781565952,"bytes_total":289910292480,"degraded_objects":28922,"degraded_total":44,"degraded_ratio":657.31818181818187,"recovering_objects_per_sec":51,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":614,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":416},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:04:36.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:04:36.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:37.026 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:37.029 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:37.039 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-09-18T12:04:35.265696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13672,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1102272,"kb_used_data":171304,"kb_used_omap":13,"kb_used_meta":930930,"kb_avail":282013248,"statfs":{"total":289910292480,"available":288781565952,"internally_reserved":0,"allocated":175415296,"data_stored":174629136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":953272850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20480024,"num_objects":-5001,"num_object_clones":0,"num_object_copies":-10002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5001,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007077"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19611,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.866144+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:30.866144+0000","last_peered":"2024-09-18T12:04:30.866144+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:30.866144+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19594,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.851728+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:30.851728+0000","last_peered":"2024-09-18T12:04:30.851728+0000","last_clean":"2024-09-18T12:04:30.851728+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:30.851728+0000","last_undegraded":"2024-09-18T12:04:30.851728+0000","last_fullsized":"2024-09-18T12:04:30.851728+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19748,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.900026+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:30.900026+0000","last_peered":"2024-09-18T12:04:30.900026+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:30.900026+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19293,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.884579+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:30.884579+0000","last_peered":"2024-09-18T12:04:30.884579+0000","last_clean":"2024-09-18T12:04:30.884579+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:30.884579+0000","last_undegraded":"2024-09-18T12:04:30.884579+0000","last_fullsized":"2024-09-18T12:04:30.884579+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19642,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.690185+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:30.690185+0000","last_peered":"2024-09-18T12:04:30.690185+0000","last_clean":"2024-09-18T12:04:30.690185+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:30.690185+0000","last_undegraded":"2024-09-18T12:04:30.690185+0000","last_fullsized":"2024-09-18T12:04:30.690185+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19508,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.850753+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:30.850753+0000","last_peered":"2024-09-18T12:04:30.850753+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:30.850753+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19463,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.794526+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:30.794526+0000","last_peered":"2024-09-18T12:04:30.794526+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:30.794526+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19831,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.833697+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:30.833697+0000","last_peered":"2024-09-18T12:04:30.833697+0000","last_clean":"2024-09-18T12:04:30.833697+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:30.833697+0000","last_undegraded":"2024-09-18T12:04:30.833697+0000","last_fullsized":"2024-09-18T12:04:30.833697+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26537,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.903687+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:30.903687+0000","last_peered":"2024-09-18T12:04:30.903687+0000","last_clean":"2024-09-18T12:04:30.903687+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:30.903687+0000","last_undegraded":"2024-09-18T12:04:30.903687+0000","last_fullsized":"2024-09-18T12:04:30.903687+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.493075+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:43.493075+0000","last_peered":"2024-09-18T12:01:43.493075+0000","last_clean":"2024-09-18T12:01:43.493075+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:43.493075+0000","last_undegraded":"2024-09-18T12:01:43.493075+0000","last_fullsized":"2024-09-18T12:01:43.493075+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":25623,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:34.378336+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:34.378336+0000","last_peered":"2024-09-18T12:04:34.378336+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:34.378336+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3049,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":19523,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.910024+0000","last_change":"2024-09-18T11:55:04.732008+0000","last_active":"2024-09-18T12:04:30.910024+0000","last_peered":"2024-09-18T12:04:30.910024+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:30.910024+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26911,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:30.872255+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:30.872255+0000","last_peered":"2024-09-18T12:04:30.872255+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:30.872255+0000","last_undegraded":"2024-09-18T12:04:30.872255+0000","last_fullsized":"2024-09-18T12:04:30.872255+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19614,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.878037+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:30.878037+0000","last_peered":"2024-09-18T12:04:30.878037+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:30.878037+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26672,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.816869+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:30.816869+0000","last_peered":"2024-09-18T12:04:30.816869+0000","last_clean":"2024-09-18T12:04:30.816869+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:30.816869+0000","last_undegraded":"2024-09-18T12:04:30.816869+0000","last_fullsized":"2024-09-18T12:04:30.816869+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":43,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.492122+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:43.492122+0000","last_peered":"2024-09-18T12:01:43.492122+0000","last_clean":"2024-09-18T12:01:43.492122+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:43.492122+0000","last_undegraded":"2024-09-18T12:01:43.492122+0000","last_fullsized":"2024-09-18T12:01:43.492122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19283,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.738824+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:30.738824+0000","last_peered":"2024-09-18T12:04:30.738824+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:30.738824+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19601,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.897773+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:30.897773+0000","last_peered":"2024-09-18T12:04:30.897773+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:30.897773+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13671,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174325760,"data_stored":174270096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051712,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420096,"kb_used_data":113912,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951744,"statfs":{"total":96636764160,"available":96206585856,"internally_reserved":0,"allocated":116645888,"data_stored":116401325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.34399999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.52700000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855474,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314916,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":94056924,"statfs":{"total":96636764160,"available":96314290176,"internally_reserved":0,"allocated":421888,"data_stored":128476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":322039302},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.311},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.34200000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367260,"kb_used_data":56980,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004580,"statfs":{"total":96636764160,"available":96260689920,"internally_reserved":0,"allocated":58347520,"data_stored":58099335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":8638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:40.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:40.357 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:40.360 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:40.371 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-18T12:04:39.267191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13795,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089804,"kb_used_data":171312,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025716,"statfs":{"total":289910292480,"available":288794333184,"internally_reserved":0,"allocated":175423488,"data_stored":174632054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3764248,"num_objects":-920,"num_object_clones":0,"num_object_copies":-1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-558,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":558,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002841"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19611,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.866144+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:30.866144+0000","last_peered":"2024-09-18T12:04:30.866144+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:30.866144+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19595,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.876292+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:30.876292+0000","last_peered":"2024-09-18T12:04:30.876292+0000","last_clean":"2024-09-18T12:04:30.876292+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:30.876292+0000","last_undegraded":"2024-09-18T12:04:30.876292+0000","last_fullsized":"2024-09-18T12:04:30.876292+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19749,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.920909+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:30.920909+0000","last_peered":"2024-09-18T12:04:30.920909+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:30.920909+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19294,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.884680+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:30.884680+0000","last_peered":"2024-09-18T12:04:30.884680+0000","last_clean":"2024-09-18T12:04:30.884680+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:30.884680+0000","last_undegraded":"2024-09-18T12:04:30.884680+0000","last_fullsized":"2024-09-18T12:04:30.884680+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19642,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.690185+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:30.690185+0000","last_peered":"2024-09-18T12:04:30.690185+0000","last_clean":"2024-09-18T12:04:30.690185+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:30.690185+0000","last_undegraded":"2024-09-18T12:04:30.690185+0000","last_fullsized":"2024-09-18T12:04:30.690185+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19508,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.850753+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:30.850753+0000","last_peered":"2024-09-18T12:04:30.850753+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:30.850753+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19463,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.794526+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:30.794526+0000","last_peered":"2024-09-18T12:04:30.794526+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:30.794526+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19831,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.833697+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:30.833697+0000","last_peered":"2024-09-18T12:04:30.833697+0000","last_clean":"2024-09-18T12:04:30.833697+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:30.833697+0000","last_undegraded":"2024-09-18T12:04:30.833697+0000","last_fullsized":"2024-09-18T12:04:30.833697+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26537,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.903687+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:30.903687+0000","last_peered":"2024-09-18T12:04:30.903687+0000","last_clean":"2024-09-18T12:04:30.903687+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:30.903687+0000","last_undegraded":"2024-09-18T12:04:30.903687+0000","last_fullsized":"2024-09-18T12:04:30.903687+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":44,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.486642+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:44.486642+0000","last_peered":"2024-09-18T12:01:44.486642+0000","last_clean":"2024-09-18T12:01:44.486642+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:44.486642+0000","last_undegraded":"2024-09-18T12:01:44.486642+0000","last_fullsized":"2024-09-18T12:01:44.486642+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":25623,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:34.378336+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:34.378336+0000","last_peered":"2024-09-18T12:04:34.378336+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:34.378336+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3049,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":19771,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:39.044747+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:39.044747+0000","last_peered":"2024-09-18T12:04:39.044747+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:39.044747+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26912,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:30.889953+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:30.889953+0000","last_peered":"2024-09-18T12:04:30.889953+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:30.889953+0000","last_undegraded":"2024-09-18T12:04:30.889953+0000","last_fullsized":"2024-09-18T12:04:30.889953+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19614,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.878037+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:30.878037+0000","last_peered":"2024-09-18T12:04:30.878037+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:30.878037+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26672,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.816869+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:30.816869+0000","last_peered":"2024-09-18T12:04:30.816869+0000","last_clean":"2024-09-18T12:04:30.816869+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:30.816869+0000","last_undegraded":"2024-09-18T12:04:30.816869+0000","last_fullsized":"2024-09-18T12:04:30.816869+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":44,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.488722+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:44.488722+0000","last_peered":"2024-09-18T12:01:44.488722+0000","last_clean":"2024-09-18T12:01:44.488722+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:44.488722+0000","last_undegraded":"2024-09-18T12:01:44.488722+0000","last_fullsized":"2024-09-18T12:01:44.488722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19283,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.738824+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:30.738824+0000","last_peered":"2024-09-18T12:04:30.738824+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:30.738824+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19601,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.897773+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:30.897773+0000","last_peered":"2024-09-18T12:04:30.897773+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:30.897773+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13794,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174325760,"data_stored":174270096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051713,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420108,"kb_used_data":113920,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951732,"statfs":{"total":96636764160,"available":96206573568,"internally_reserved":0,"allocated":116654080,"data_stored":116404243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.54900000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855475,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302436,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069404,"statfs":{"total":96636764160,"available":96327069696,"internally_reserved":0,"allocated":421888,"data_stored":128476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.5}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.54000000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367260,"kb_used_data":56980,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004580,"statfs":{"total":96636764160,"available":96260689920,"internally_reserved":0,"allocated":58347520,"data_stored":58099335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":8638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:40.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:40.669 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:40.671 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:40.684 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-18T12:04:39.267191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13795,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089804,"kb_used_data":171312,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025716,"statfs":{"total":289910292480,"available":288794333184,"internally_reserved":0,"allocated":175423488,"data_stored":174632054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3764248,"num_objects":-920,"num_object_clones":0,"num_object_copies":-1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-558,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":558,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002841"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19611,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.866144+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:30.866144+0000","last_peered":"2024-09-18T12:04:30.866144+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:30.866144+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19595,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.876292+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:30.876292+0000","last_peered":"2024-09-18T12:04:30.876292+0000","last_clean":"2024-09-18T12:04:30.876292+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:30.876292+0000","last_undegraded":"2024-09-18T12:04:30.876292+0000","last_fullsized":"2024-09-18T12:04:30.876292+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19749,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.920909+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:30.920909+0000","last_peered":"2024-09-18T12:04:30.920909+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:30.920909+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19294,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.884680+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:30.884680+0000","last_peered":"2024-09-18T12:04:30.884680+0000","last_clean":"2024-09-18T12:04:30.884680+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:30.884680+0000","last_undegraded":"2024-09-18T12:04:30.884680+0000","last_fullsized":"2024-09-18T12:04:30.884680+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19642,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.690185+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:30.690185+0000","last_peered":"2024-09-18T12:04:30.690185+0000","last_clean":"2024-09-18T12:04:30.690185+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:30.690185+0000","last_undegraded":"2024-09-18T12:04:30.690185+0000","last_fullsized":"2024-09-18T12:04:30.690185+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19508,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.850753+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:30.850753+0000","last_peered":"2024-09-18T12:04:30.850753+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:30.850753+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19463,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.794526+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:30.794526+0000","last_peered":"2024-09-18T12:04:30.794526+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:30.794526+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19831,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.833697+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:30.833697+0000","last_peered":"2024-09-18T12:04:30.833697+0000","last_clean":"2024-09-18T12:04:30.833697+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:30.833697+0000","last_undegraded":"2024-09-18T12:04:30.833697+0000","last_fullsized":"2024-09-18T12:04:30.833697+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26537,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.903687+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:30.903687+0000","last_peered":"2024-09-18T12:04:30.903687+0000","last_clean":"2024-09-18T12:04:30.903687+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:30.903687+0000","last_undegraded":"2024-09-18T12:04:30.903687+0000","last_fullsized":"2024-09-18T12:04:30.903687+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":44,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.486642+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:44.486642+0000","last_peered":"2024-09-18T12:01:44.486642+0000","last_clean":"2024-09-18T12:01:44.486642+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:44.486642+0000","last_undegraded":"2024-09-18T12:01:44.486642+0000","last_fullsized":"2024-09-18T12:01:44.486642+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":25623,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:34.378336+0000","last_change":"2024-09-18T12:01:42.966902+0000","last_active":"2024-09-18T12:04:34.378336+0000","last_peered":"2024-09-18T12:04:34.378336+0000","last_clean":"2024-09-18T11:55:02.220264+0000","last_became_active":"2024-09-18T11:55:04.718895+0000","last_became_peered":"2024-09-18T11:55:04.718895+0000","last_unstale":"2024-09-18T12:04:34.378336+0000","last_undegraded":"2024-09-18T11:55:04.327111+0000","last_fullsized":"2024-09-18T11:55:04.249084+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":487,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3049,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":95,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.487221+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:01:43.487221+0000","last_peered":"2024-09-18T12:01:43.487221+0000","last_clean":"2024-09-18T12:01:43.487221+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:01:43.487221+0000","last_undegraded":"2024-09-18T12:01:43.487221+0000","last_fullsized":"2024-09-18T12:01:43.487221+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":19771,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:39.044747+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:39.044747+0000","last_peered":"2024-09-18T12:04:39.044747+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:39.044747+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26912,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:30.889953+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:30.889953+0000","last_peered":"2024-09-18T12:04:30.889953+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:30.889953+0000","last_undegraded":"2024-09-18T12:04:30.889953+0000","last_fullsized":"2024-09-18T12:04:30.889953+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19614,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.878037+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:30.878037+0000","last_peered":"2024-09-18T12:04:30.878037+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:30.878037+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26672,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.816869+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:30.816869+0000","last_peered":"2024-09-18T12:04:30.816869+0000","last_clean":"2024-09-18T12:04:30.816869+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:30.816869+0000","last_undegraded":"2024-09-18T12:04:30.816869+0000","last_fullsized":"2024-09-18T12:04:30.816869+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":44,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.488722+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:44.488722+0000","last_peered":"2024-09-18T12:01:44.488722+0000","last_clean":"2024-09-18T12:01:44.488722+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:44.488722+0000","last_undegraded":"2024-09-18T12:01:44.488722+0000","last_fullsized":"2024-09-18T12:01:44.488722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19283,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.738824+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:30.738824+0000","last_peered":"2024-09-18T12:04:30.738824+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:30.738824+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19601,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.897773+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:30.897773+0000","last_peered":"2024-09-18T12:04:30.897773+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:30.897773+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13794,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174325760,"data_stored":174270096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051713,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420108,"kb_used_data":113920,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951732,"statfs":{"total":96636764160,"available":96206573568,"internally_reserved":0,"allocated":116654080,"data_stored":116404243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.54900000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855475,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302436,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069404,"statfs":{"total":96636764160,"available":96327069696,"internally_reserved":0,"allocated":421888,"data_stored":128476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.5}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.54000000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367260,"kb_used_data":56980,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004580,"statfs":{"total":96636764160,"available":96260689920,"internally_reserved":0,"allocated":58347520,"data_stored":58099335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:03:34 2024","interfaces":[{"interface":"back","average":{"1min":1.78,"5min":1.857,"15min":1.099},"min":{"1min":0.223,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":11.071999999999999,"5min":13.773,"15min":13.773},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.464,"15min":0.86899999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":7.2320000000000002,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":8638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:40.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:04:40.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19611, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.866144+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:30.866144+0000', 'last_peered': '2024-09-18T12:04:30.866144+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:30.866144+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:40.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:04:40.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19749, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.920909+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:30.920909+0000', 'last_peered': '2024-09-18T12:04:30.920909+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:30.920909+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:40.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:04:40.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19508, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.850753+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:30.850753+0000', 'last_peered': '2024-09-18T12:04:30.850753+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:30.850753+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:40.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:04:40.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19463, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.794526+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:30.794526+0000', 'last_peered': '2024-09-18T12:04:30.794526+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:30.794526+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:40.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-18T12:04:40.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "43'19446", 'reported_seq': 25623, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:34.378336+0000', 'last_change': '2024-09-18T12:01:42.966902+0000', 'last_active': '2024-09-18T12:04:34.378336+0000', 'last_peered': '2024-09-18T12:04:34.378336+0000', 'last_clean': '2024-09-18T11:55:02.220264+0000', 'last_became_active': '2024-09-18T11:55:04.718895+0000', 'last_became_peered': '2024-09-18T11:55:04.718895+0000', 'last_unstale': '2024-09-18T12:04:34.378336+0000', 'last_undegraded': '2024-09-18T11:55:04.327111+0000', 'last_fullsized': '2024-09-18T11:55:04.249084+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 487, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19444, 'num_write_kb': 38880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3049, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -485}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:40.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:04:40.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19463", 'reported_seq': 19771, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:39.044747+0000', 'last_change': '2024-09-18T12:04:38.127143+0000', 'last_active': '2024-09-18T12:04:39.044747+0000', 'last_peered': '2024-09-18T12:04:39.044747+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:04:39.044747+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3429, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19461, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 125, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3427}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:40.687 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:04:40.687 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26912, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:30.889953+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:30.889953+0000', 'last_peered': '2024-09-18T12:04:30.889953+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:30.889953+0000', 'last_undegraded': '2024-09-18T12:04:30.889953+0000', 'last_fullsized': '2024-09-18T12:04:30.889953+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:40.687 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:04:40.687 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19614, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.878037+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:30.878037+0000', 'last_peered': '2024-09-18T12:04:30.878037+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:30.878037+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:40.687 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:04:40.687 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 19283, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.738824+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:30.738824+0000', 'last_peered': '2024-09-18T12:04:30.738824+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:30.738824+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:40.687 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:04:40.687 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19601, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.897773+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:30.897773+0000', 'last_peered': '2024-09-18T12:04:30.897773+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:30.897773+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:40.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:04:41.046 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:41.046 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28799/44 objects degraded (65452.273%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":681,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":2},{"state_name":"active+recovery_wait","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1115959296,"bytes_avail":288794333184,"bytes_total":289910292480,"degraded_objects":28799,"degraded_total":44,"degraded_ratio":654.52272727272725,"recovering_objects_per_sec":69,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":558,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":114},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:04:41.059 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:04:41.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:41.342 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:41.344 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:41.353 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-09-18T12:04:41.267754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14282,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089828,"kb_used_data":171328,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025692,"statfs":{"total":289910292480,"available":288794308608,"internally_reserved":0,"allocated":175439872,"data_stored":174637869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2670616,"num_objects":-653,"num_object_clones":0,"num_object_copies":-1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":653,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002659"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19595,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.876292+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:30.876292+0000","last_peered":"2024-09-18T12:04:30.876292+0000","last_clean":"2024-09-18T12:04:30.876292+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:30.876292+0000","last_undegraded":"2024-09-18T12:04:30.876292+0000","last_fullsized":"2024-09-18T12:04:30.876292+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19749,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.920909+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:30.920909+0000","last_peered":"2024-09-18T12:04:30.920909+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:30.920909+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19294,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.884680+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:30.884680+0000","last_peered":"2024-09-18T12:04:30.884680+0000","last_clean":"2024-09-18T12:04:30.884680+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:30.884680+0000","last_undegraded":"2024-09-18T12:04:30.884680+0000","last_fullsized":"2024-09-18T12:04:30.884680+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495793+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:01:43.495793+0000","last_peered":"2024-09-18T12:01:43.495793+0000","last_clean":"2024-09-18T12:01:43.495793+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:01:43.495793+0000","last_undegraded":"2024-09-18T12:01:43.495793+0000","last_fullsized":"2024-09-18T12:01:43.495793+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26537,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.903687+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:30.903687+0000","last_peered":"2024-09-18T12:04:30.903687+0000","last_clean":"2024-09-18T12:04:30.903687+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:30.903687+0000","last_undegraded":"2024-09-18T12:04:30.903687+0000","last_fullsized":"2024-09-18T12:04:30.903687+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":44,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.486642+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:44.486642+0000","last_peered":"2024-09-18T12:01:44.486642+0000","last_clean":"2024-09-18T12:01:44.486642+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:44.486642+0000","last_undegraded":"2024-09-18T12:01:44.486642+0000","last_fullsized":"2024-09-18T12:01:44.486642+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":19771,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:39.044747+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:39.044747+0000","last_peered":"2024-09-18T12:04:39.044747+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:39.044747+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":75,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495852+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:01:43.495852+0000","last_peered":"2024-09-18T12:01:43.495852+0000","last_clean":"2024-09-18T12:01:43.495852+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:01:43.495852+0000","last_undegraded":"2024-09-18T12:01:43.495852+0000","last_fullsized":"2024-09-18T12:01:43.495852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26912,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:30.889953+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:30.889953+0000","last_peered":"2024-09-18T12:04:30.889953+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:30.889953+0000","last_undegraded":"2024-09-18T12:04:30.889953+0000","last_fullsized":"2024-09-18T12:04:30.889953+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.519496+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:01:43.519496+0000","last_peered":"2024-09-18T12:01:43.519496+0000","last_clean":"2024-09-18T12:01:43.519496+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:01:43.519496+0000","last_undegraded":"2024-09-18T12:01:43.519496+0000","last_fullsized":"2024-09-18T12:01:43.519496+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26672,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.816869+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:30.816869+0000","last_peered":"2024-09-18T12:04:30.816869+0000","last_clean":"2024-09-18T12:04:30.816869+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:30.816869+0000","last_undegraded":"2024-09-18T12:04:30.816869+0000","last_fullsized":"2024-09-18T12:04:30.816869+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":44,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.488722+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:44.488722+0000","last_peered":"2024-09-18T12:01:44.488722+0000","last_clean":"2024-09-18T12:01:44.488722+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:44.488722+0000","last_undegraded":"2024-09-18T12:01:44.488722+0000","last_fullsized":"2024-09-18T12:01:44.488722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":59,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.495874+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:01:43.495874+0000","last_peered":"2024-09-18T12:01:43.495874+0000","last_clean":"2024-09-18T12:01:43.495874+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:01:43.495874+0000","last_undegraded":"2024-09-18T12:01:43.495874+0000","last_fullsized":"2024-09-18T12:01:43.495874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T12:01:43.498134+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:01:43.498134+0000","last_peered":"2024-09-18T12:01:43.498134+0000","last_clean":"2024-09-18T12:01:43.498134+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:01:43.498134+0000","last_undegraded":"2024-09-18T12:01:43.498134+0000","last_fullsized":"2024-09-18T12:01:43.498134+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14281,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174325760,"data_stored":174270096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051713,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420108,"kb_used_data":113920,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951732,"statfs":{"total":96636764160,"available":96206573568,"internally_reserved":0,"allocated":116654080,"data_stored":116404243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.54900000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855475,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302436,"kb_used_data":412,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069404,"statfs":{"total":96636764160,"available":96327069696,"internally_reserved":0,"allocated":421888,"data_stored":128476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.5}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.54000000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.247},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.56000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":8638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:44.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:44.670 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:44.681 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-18T12:04:43.268226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14282,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1093996,"kb_used_data":171336,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021524,"statfs":{"total":289910292480,"available":288790040576,"internally_reserved":0,"allocated":175448064,"data_stored":174642450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":610,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002529"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19595,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.876292+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:30.876292+0000","last_peered":"2024-09-18T12:04:30.876292+0000","last_clean":"2024-09-18T12:04:30.876292+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:30.876292+0000","last_undegraded":"2024-09-18T12:04:30.876292+0000","last_fullsized":"2024-09-18T12:04:30.876292+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19749,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.920909+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:30.920909+0000","last_peered":"2024-09-18T12:04:30.920909+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:30.920909+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19294,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.884680+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:30.884680+0000","last_peered":"2024-09-18T12:04:30.884680+0000","last_clean":"2024-09-18T12:04:30.884680+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:30.884680+0000","last_undegraded":"2024-09-18T12:04:30.884680+0000","last_fullsized":"2024-09-18T12:04:30.884680+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":44,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.486642+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:44.486642+0000","last_peered":"2024-09-18T12:01:44.486642+0000","last_clean":"2024-09-18T12:01:44.486642+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:44.486642+0000","last_undegraded":"2024-09-18T12:01:44.486642+0000","last_fullsized":"2024-09-18T12:01:44.486642+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":19771,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:39.044747+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:39.044747+0000","last_peered":"2024-09-18T12:04:39.044747+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:39.044747+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26912,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:30.889953+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:30.889953+0000","last_peered":"2024-09-18T12:04:30.889953+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:30.889953+0000","last_undegraded":"2024-09-18T12:04:30.889953+0000","last_fullsized":"2024-09-18T12:04:30.889953+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":44,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.488722+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:44.488722+0000","last_peered":"2024-09-18T12:01:44.488722+0000","last_clean":"2024-09-18T12:01:44.488722+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:44.488722+0000","last_undegraded":"2024-09-18T12:01:44.488722+0000","last_fullsized":"2024-09-18T12:01:44.488722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14281,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051713,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420108,"kb_used_data":113920,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951732,"statfs":{"total":96636764160,"available":96206573568,"internally_reserved":0,"allocated":116654080,"data_stored":116404243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.54900000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855476,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306604,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065236,"statfs":{"total":96636764160,"available":96322801664,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.65100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.247},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.56000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:44.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:44.980 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:44.982 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:44.992 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-18T12:04:43.268226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14282,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1093996,"kb_used_data":171336,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021524,"statfs":{"total":289910292480,"available":288790040576,"internally_reserved":0,"allocated":175448064,"data_stored":174642450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":610,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002529"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19595,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.876292+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:30.876292+0000","last_peered":"2024-09-18T12:04:30.876292+0000","last_clean":"2024-09-18T12:04:30.876292+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:30.876292+0000","last_undegraded":"2024-09-18T12:04:30.876292+0000","last_fullsized":"2024-09-18T12:04:30.876292+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19749,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:30.920909+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:30.920909+0000","last_peered":"2024-09-18T12:04:30.920909+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:30.920909+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19294,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:04:30.884680+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:30.884680+0000","last_peered":"2024-09-18T12:04:30.884680+0000","last_clean":"2024-09-18T12:04:30.884680+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:30.884680+0000","last_undegraded":"2024-09-18T12:04:30.884680+0000","last_fullsized":"2024-09-18T12:04:30.884680+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":44,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.486642+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:01:44.486642+0000","last_peered":"2024-09-18T12:01:44.486642+0000","last_clean":"2024-09-18T12:01:44.486642+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:01:44.486642+0000","last_undegraded":"2024-09-18T12:01:44.486642+0000","last_fullsized":"2024-09-18T12:01:44.486642+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":19771,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:39.044747+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:39.044747+0000","last_peered":"2024-09-18T12:04:39.044747+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:39.044747+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26912,"reported_epoch":43,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:30.889953+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:30.889953+0000","last_peered":"2024-09-18T12:04:30.889953+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:30.889953+0000","last_undegraded":"2024-09-18T12:04:30.889953+0000","last_fullsized":"2024-09-18T12:04:30.889953+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":44,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T12:01:44.488722+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:01:44.488722+0000","last_peered":"2024-09-18T12:01:44.488722+0000","last_clean":"2024-09-18T12:01:44.488722+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:01:44.488722+0000","last_undegraded":"2024-09-18T12:01:44.488722+0000","last_fullsized":"2024-09-18T12:01:44.488722+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14281,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051713,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420108,"kb_used_data":113920,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951732,"statfs":{"total":96636764160,"available":96206573568,"internally_reserved":0,"allocated":116654080,"data_stored":116404243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.54900000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855476,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306604,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065236,"statfs":{"total":96636764160,"available":96322801664,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.65100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.247},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.56000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:44.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:04:44.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19613, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156287+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:38.156287+0000', 'last_peered': '2024-09-18T12:04:38.156287+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:38.156287+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:44.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:04:44.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19749, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:30.920909+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:30.920909+0000', 'last_peered': '2024-09-18T12:04:30.920909+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:30.920909+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:44.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:04:44.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19510, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156169+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:38.156169+0000', 'last_peered': '2024-09-18T12:04:38.156169+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:38.156169+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:44.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:04:44.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19465, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157734+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:38.157734+0000', 'last_peered': '2024-09-18T12:04:38.157734+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:38.157734+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:44.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:04:44.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19463", 'reported_seq': 19771, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:39.044747+0000', 'last_change': '2024-09-18T12:04:38.127143+0000', 'last_active': '2024-09-18T12:04:39.044747+0000', 'last_peered': '2024-09-18T12:04:39.044747+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:04:39.044747+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3429, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19461, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 125, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3427}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:44.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:04:44.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26912, 'reported_epoch': 43, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:30.889953+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:30.889953+0000', 'last_peered': '2024-09-18T12:04:30.889953+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:30.889953+0000', 'last_undegraded': '2024-09-18T12:04:30.889953+0000', 'last_fullsized': '2024-09-18T12:04:30.889953+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:44.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:04:44.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19616, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156213+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:38.156213+0000', 'last_peered': '2024-09-18T12:04:38.156213+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:38.156213+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:44.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:04:44.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 19285, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157224+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:38.157224+0000', 'last_peered': '2024-09-18T12:04:38.157224+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:38.157224+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:44.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:04:44.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19603, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156505+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:38.156505+0000', 'last_peered': '2024-09-18T12:04:38.156505+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:38.156505+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:44.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:04:45.358 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:45.358 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27965/44 objects degraded (63556.818%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":685,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1120264192,"bytes_avail":288790028288,"bytes_total":289910292480,"degraded_objects":27965,"degraded_total":44,"degraded_ratio":635.56818181818187,"recovering_objects_per_sec":119,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":957,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:04:45.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:04:45.376 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:45.654 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:45.656 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:45.671 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-18T12:04:45.268871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14629,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094008,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021512,"statfs":{"total":289910292480,"available":288790028288,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002376"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":20466,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:44.051418+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:44.051418+0000","last_peered":"2024-09-18T12:04:44.051418+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:44.051418+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3080}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14628,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051714,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.436},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33100000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.378},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.38800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855476,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306604,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065236,"statfs":{"total":96636764160,"available":96322801664,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.65100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.30099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.38},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:48.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:48.991 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:48.994 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:49.004 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-09-18T12:04:47.269594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14629,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098168,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017352,"statfs":{"total":289910292480,"available":288785768448,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":957,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.116799"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":20466,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:44.051418+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:44.051418+0000","last_peered":"2024-09-18T12:04:44.051418+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:44.051418+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3080}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14628,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051714,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.436},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33100000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.378},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.38800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855477,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310764,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061076,"statfs":{"total":96636764160,"available":96318541824,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.437},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.57299999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.30099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.38},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:49.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:49.312 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:49.315 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:49.325 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-09-18T12:04:49.270423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14992,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098168,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017352,"statfs":{"total":289910292480,"available":288785768448,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.120401"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":21192,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:49.051693+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:49.051693+0000","last_peered":"2024-09-18T12:04:49.051693+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:49.051693+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":835,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2717}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14991,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051715,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.50600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.54400000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855477,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310764,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061076,"statfs":{"total":96636764160,"available":96318541824,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.437},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.57299999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.30099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.38},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:49.326 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:04:49.327 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19613, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156287+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:38.156287+0000', 'last_peered': '2024-09-18T12:04:38.156287+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:38.156287+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:49.327 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:04:49.327 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19750, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.160937+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:38.160937+0000', 'last_peered': '2024-09-18T12:04:38.160937+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:38.160937+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:49.327 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:04:49.327 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19510, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156169+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:38.156169+0000', 'last_peered': '2024-09-18T12:04:38.156169+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:38.156169+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:49.328 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:04:49.328 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19465, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157734+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:38.157734+0000', 'last_peered': '2024-09-18T12:04:38.157734+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:38.157734+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:49.328 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:04:49.328 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19463", 'reported_seq': 21192, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:49.051693+0000', 'last_change': '2024-09-18T12:04:38.127143+0000', 'last_active': '2024-09-18T12:04:49.051693+0000', 'last_peered': '2024-09-18T12:04:49.051693+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:04:49.051693+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2719, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19461, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 835, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2717}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:49.328 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:04:49.328 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26913, 'reported_epoch': 44, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:38.159248+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:38.159248+0000', 'last_peered': '2024-09-18T12:04:38.159248+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:38.159248+0000', 'last_undegraded': '2024-09-18T12:04:38.159248+0000', 'last_fullsized': '2024-09-18T12:04:38.159248+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:49.328 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:04:49.328 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19616, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156213+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:38.156213+0000', 'last_peered': '2024-09-18T12:04:38.156213+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:38.156213+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:49.329 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:04:49.329 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 19285, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157224+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:38.157224+0000', 'last_peered': '2024-09-18T12:04:38.157224+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:38.157224+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:49.329 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:04:49.329 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19603, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156505+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:38.156505+0000', 'last_peered': '2024-09-18T12:04:38.156505+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:38.156505+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:49.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:04:49.726 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:49.726 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27602/44 objects degraded (62731.818%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":689,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1124524032,"bytes_avail":288785768448,"bytes_total":289910292480,"degraded_objects":27602,"degraded_total":44,"degraded_ratio":627.31818181818187,"recovering_objects_per_sec":118,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1197,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:04:49.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:04:49.743 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:50.037 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:50.048 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-09-18T12:04:49.270423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14992,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098168,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017352,"statfs":{"total":289910292480,"available":288785768448,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.120401"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":21192,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:49.051693+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:49.051693+0000","last_peered":"2024-09-18T12:04:49.051693+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:49.051693+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":835,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2717}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14991,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051715,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.50600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.54400000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855477,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310764,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061076,"statfs":{"total":96636764160,"available":96318541824,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.437},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.57299999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.30099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.38},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:53.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:54.742 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:54.753 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-09-18T12:04:53.271842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14992,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098168,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017352,"statfs":{"total":289910292480,"available":288785768448,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004087"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":21192,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:49.051693+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:49.051693+0000","last_peered":"2024-09-18T12:04:49.051693+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:49.051693+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":835,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2717}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14991,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051715,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.50600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.54400000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855478,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310764,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061076,"statfs":{"total":96636764160,"available":96318541824,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.313}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.41699999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.45300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.57299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:54.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:55.091 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:55.094 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:55.104 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-09-18T12:04:53.271842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14992,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098168,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017352,"statfs":{"total":289910292480,"available":288785768448,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004087"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":21192,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:49.051693+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:49.051693+0000","last_peered":"2024-09-18T12:04:49.051693+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:49.051693+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":835,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2717}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27602,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14991,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051715,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.50600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.54400000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855478,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310764,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061076,"statfs":{"total":96636764160,"available":96318541824,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.313}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.41699999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.45300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.57299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:55.106 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:04:55.106 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19613, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156287+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:38.156287+0000', 'last_peered': '2024-09-18T12:04:38.156287+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:38.156287+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:55.106 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:04:55.106 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19750, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.160937+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:38.160937+0000', 'last_peered': '2024-09-18T12:04:38.160937+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:38.160937+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:55.106 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:04:55.106 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19510, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156169+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:38.156169+0000', 'last_peered': '2024-09-18T12:04:38.156169+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:38.156169+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:55.107 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:04:55.107 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19465, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157734+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:38.157734+0000', 'last_peered': '2024-09-18T12:04:38.157734+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:38.157734+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:55.107 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:04:55.107 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19463", 'reported_seq': 21192, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:49.051693+0000', 'last_change': '2024-09-18T12:04:38.127143+0000', 'last_active': '2024-09-18T12:04:49.051693+0000', 'last_peered': '2024-09-18T12:04:49.051693+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:04:49.051693+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2719, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19461, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 835, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2717}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:55.107 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:04:55.107 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26913, 'reported_epoch': 44, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:38.159248+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:38.159248+0000', 'last_peered': '2024-09-18T12:04:38.159248+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:38.159248+0000', 'last_undegraded': '2024-09-18T12:04:38.159248+0000', 'last_fullsized': '2024-09-18T12:04:38.159248+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:55.107 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:04:55.107 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19616, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156213+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:38.156213+0000', 'last_peered': '2024-09-18T12:04:38.156213+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:38.156213+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:55.107 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:04:55.107 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 19285, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157224+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:38.157224+0000', 'last_peered': '2024-09-18T12:04:38.157224+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:38.157224+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:55.108 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:04:55.108 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19603, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156505+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:38.156505+0000', 'last_peered': '2024-09-18T12:04:38.156505+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:38.156505+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:55.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:04:55.453 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:55.453 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27225/44 objects degraded (61875.000%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":695,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1124524032,"bytes_avail":288785768448,"bytes_total":289910292480,"degraded_objects":27225,"degraded_total":44,"degraded_ratio":618.75,"recovering_objects_per_sec":90,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1087,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:04:55.467 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:04:55.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:55.853 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:55.866 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-18T12:04:55.272407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15369,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098168,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017352,"statfs":{"total":289910292480,"available":288785768448,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004181"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":21946,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:54.053186+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:54.053186+0000","last_peered":"2024-09-18T12:04:54.053186+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:54.053186+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1212,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2340}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15368,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051716,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.48899999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.45300000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855478,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310764,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061076,"statfs":{"total":96636764160,"available":96318541824,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.313}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.41699999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:58.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:59.253 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:59.255 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:59.278 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-18T12:04:57.272961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15369,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1085688,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":282029832,"statfs":{"total":289910292480,"available":288798547968,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004090"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":21946,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:54.053186+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:54.053186+0000","last_peered":"2024-09-18T12:04:54.053186+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:54.053186+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1212,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2340}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15368,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051716,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.48899999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.45300000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855479,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298284,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073556,"statfs":{"total":96636764160,"available":96331321344,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.308},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.39800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:59.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:04:59.600 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:59.603 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:04:59.615 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-18T12:04:59.273709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15774,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1085688,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":282029832,"statfs":{"total":289910292480,"available":288798547968,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004114"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":22756,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:59.051817+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:59.051817+0000","last_peered":"2024-09-18T12:04:59.051817+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:59.051817+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1935}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15773,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051717,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.439},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.37}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.40000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855479,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298284,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073556,"statfs":{"total":96636764160,"available":96331321344,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.308},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.39800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:04:59.617 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:04:59.617 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19613, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156287+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:38.156287+0000', 'last_peered': '2024-09-18T12:04:38.156287+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:38.156287+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:59.617 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:04:59.617 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19750, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.160937+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:38.160937+0000', 'last_peered': '2024-09-18T12:04:38.160937+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:38.160937+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:59.617 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:04:59.617 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19510, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156169+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:38.156169+0000', 'last_peered': '2024-09-18T12:04:38.156169+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:38.156169+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:59.618 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:04:59.618 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19465, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157734+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:38.157734+0000', 'last_peered': '2024-09-18T12:04:38.157734+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:38.157734+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:59.618 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:04:59.618 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19463", 'reported_seq': 22756, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:59.051817+0000', 'last_change': '2024-09-18T12:04:38.127143+0000', 'last_active': '2024-09-18T12:04:59.051817+0000', 'last_peered': '2024-09-18T12:04:59.051817+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:04:59.051817+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19461, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1617, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1935}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:59.618 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:04:59.618 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26913, 'reported_epoch': 44, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:38.159248+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:38.159248+0000', 'last_peered': '2024-09-18T12:04:38.159248+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:38.159248+0000', 'last_undegraded': '2024-09-18T12:04:38.159248+0000', 'last_fullsized': '2024-09-18T12:04:38.159248+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:04:59.618 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:04:59.618 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19616, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156213+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:38.156213+0000', 'last_peered': '2024-09-18T12:04:38.156213+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:38.156213+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:59.618 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:04:59.618 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 19285, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157224+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:38.157224+0000', 'last_peered': '2024-09-18T12:04:38.157224+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:38.157224+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:59.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:04:59.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19603, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156505+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:38.156505+0000', 'last_peered': '2024-09-18T12:04:38.156505+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:38.156505+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:04:59.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:04:59.972 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:04:59.972 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26820/44 objects degraded (60954.545%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":699,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1111744512,"bytes_avail":288798547968,"bytes_total":289910292480,"degraded_objects":26820,"degraded_total":44,"degraded_ratio":609.5454545454545,"recovering_objects_per_sec":95,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1145,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:04:59.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:04:59.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:00.299 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:00.301 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:00.312 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-18T12:04:59.273709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15774,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1085688,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":282029832,"statfs":{"total":289910292480,"available":288798547968,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004114"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":22756,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:59.051817+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:59.051817+0000","last_peered":"2024-09-18T12:04:59.051817+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:59.051817+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1935}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15773,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051717,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.439},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.37}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.40000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855479,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298284,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073556,"statfs":{"total":96636764160,"available":96331321344,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.308},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.39800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:03.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:03.625 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:03.627 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:03.656 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-18T12:05:03.275009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15774,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089848,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025672,"statfs":{"total":289910292480,"available":288794288128,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":782,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003892"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":22756,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:59.051817+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:59.051817+0000","last_peered":"2024-09-18T12:04:59.051817+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:59.051817+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1935}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15773,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051717,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.439},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.37}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.40000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855480,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302444,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069396,"statfs":{"total":96636764160,"available":96327061504,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.51900000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.434},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.47499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.42899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:03.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:04.029 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:04.032 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:04.041 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-18T12:05:03.275009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15774,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089848,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025672,"statfs":{"total":289910292480,"available":288794288128,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":782,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003892"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":22756,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:59.051817+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:59.051817+0000","last_peered":"2024-09-18T12:04:59.051817+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:59.051817+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1935}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15773,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051717,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.439},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.37}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.40000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855480,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302444,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069396,"statfs":{"total":96636764160,"available":96327061504,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.51900000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.434},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.47499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.42899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:04.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:04.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19613, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156287+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:38.156287+0000', 'last_peered': '2024-09-18T12:04:38.156287+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:38.156287+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:04.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:04.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19750, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.160937+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:38.160937+0000', 'last_peered': '2024-09-18T12:04:38.160937+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:38.160937+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:04.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:04.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19510, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156169+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:38.156169+0000', 'last_peered': '2024-09-18T12:04:38.156169+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:38.156169+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:04.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:04.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19465, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157734+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:38.157734+0000', 'last_peered': '2024-09-18T12:04:38.157734+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:38.157734+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:04.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:05:04.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19463", 'reported_seq': 22756, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:59.051817+0000', 'last_change': '2024-09-18T12:04:38.127143+0000', 'last_active': '2024-09-18T12:04:59.051817+0000', 'last_peered': '2024-09-18T12:04:59.051817+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:04:59.051817+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1937, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19461, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1617, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1935}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:04.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:04.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26913, 'reported_epoch': 44, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:38.159248+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:38.159248+0000', 'last_peered': '2024-09-18T12:04:38.159248+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:38.159248+0000', 'last_undegraded': '2024-09-18T12:04:38.159248+0000', 'last_fullsized': '2024-09-18T12:04:38.159248+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:04.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:04.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19616, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156213+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:38.156213+0000', 'last_peered': '2024-09-18T12:04:38.156213+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:38.156213+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:04.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:05:04.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 19285, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157224+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:38.157224+0000', 'last_peered': '2024-09-18T12:04:38.157224+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:38.157224+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:04.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:04.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19603, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156505+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:38.156505+0000', 'last_peered': '2024-09-18T12:04:38.156505+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:38.156505+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:04.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:04.391 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:04.392 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26820/44 objects degraded (60954.545%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":704,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1116004352,"bytes_avail":288794288128,"bytes_total":289910292480,"degraded_objects":26820,"degraded_total":44,"degraded_ratio":609.5454545454545,"recovering_objects_per_sec":65,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":782,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:04.405 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:04.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:04.721 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:04.725 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:04.738 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-18T12:05:03.275009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15774,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089848,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025672,"statfs":{"total":289910292480,"available":288794288128,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":782,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003892"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":22756,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:59.051817+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:04:59.051817+0000","last_peered":"2024-09-18T12:04:59.051817+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:04:59.051817+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1935}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15773,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051717,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.439},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.37}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.40000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855480,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302444,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069396,"statfs":{"total":96636764160,"available":96327061504,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.51900000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.434},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.47499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.42899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:07.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:08.041 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:08.043 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:08.058 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-18T12:05:07.276270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16221,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094008,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021512,"statfs":{"total":289910292480,"available":288790028288,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":852,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003862"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":23650,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:04.053959+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:05:04.053959+0000","last_peered":"2024-09-18T12:05:04.053959+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:05:04.053959+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2064,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1488}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16220,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051718,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.498}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.44500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855481,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306604,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065236,"statfs":{"total":96636764160,"available":96322801664,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.443}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.46999999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.372}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:08.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:08.343 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:08.356 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-18T12:05:07.276270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16221,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094008,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021512,"statfs":{"total":289910292480,"available":288790028288,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":852,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003862"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":23650,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:04.053959+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:05:04.053959+0000","last_peered":"2024-09-18T12:05:04.053959+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:05:04.053959+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2064,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1488}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16220,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051718,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.498}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.44500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855481,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306604,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065236,"statfs":{"total":96636764160,"available":96322801664,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.443}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.46999999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.372}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:08.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:08.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19613, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156287+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:38.156287+0000', 'last_peered': '2024-09-18T12:04:38.156287+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:38.156287+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:08.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:08.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19750, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.160937+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:38.160937+0000', 'last_peered': '2024-09-18T12:04:38.160937+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:38.160937+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:08.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19510, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156169+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:38.156169+0000', 'last_peered': '2024-09-18T12:04:38.156169+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:38.156169+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19465, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157734+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:38.157734+0000', 'last_peered': '2024-09-18T12:04:38.157734+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:38.157734+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:05:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19463", 'reported_seq': 23650, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:04.053959+0000', 'last_change': '2024-09-18T12:04:38.127143+0000', 'last_active': '2024-09-18T12:05:04.053959+0000', 'last_peered': '2024-09-18T12:05:04.053959+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:05:04.053959+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1490, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19461, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2064, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1488}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26913, 'reported_epoch': 44, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:38.159248+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:38.159248+0000', 'last_peered': '2024-09-18T12:04:38.159248+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:38.159248+0000', 'last_undegraded': '2024-09-18T12:04:38.159248+0000', 'last_fullsized': '2024-09-18T12:04:38.159248+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:08.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19616, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156213+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:38.156213+0000', 'last_peered': '2024-09-18T12:04:38.156213+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:38.156213+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:05:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 19285, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157224+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:38.157224+0000', 'last_peered': '2024-09-18T12:04:38.157224+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:38.157224+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:08.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19603, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156505+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:38.156505+0000', 'last_peered': '2024-09-18T12:04:38.156505+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:38.156505+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:08.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:08.719 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:08.720 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26373/44 objects degraded (59938.636%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":708,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1120264192,"bytes_avail":288790028288,"bytes_total":289910292480,"degraded_objects":26373,"degraded_total":44,"degraded_ratio":599.38636363636363,"recovering_objects_per_sec":70,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":852,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:08.738 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:08.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:09.016 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:09.018 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:09.029 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-18T12:05:07.276270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16221,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094008,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021512,"statfs":{"total":289910292480,"available":288790028288,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":852,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003862"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":23650,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:04.053959+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:05:04.053959+0000","last_peered":"2024-09-18T12:05:04.053959+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:05:04.053959+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2064,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1488}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16220,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051718,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.498}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.44500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855481,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306604,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065236,"statfs":{"total":96636764160,"available":96322801664,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.443}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.46999999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.372}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:12.032 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:12.337 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:12.340 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:12.351 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-09-18T12:05:11.277397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16718,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094008,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021512,"statfs":{"total":289910292480,"available":288790028288,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":24644,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:09.054245+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:05:09.054245+0000","last_peered":"2024-09-18T12:05:09.054245+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:05:09.054245+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2561,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-991}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16717,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051719,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.498}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.44500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855481,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306604,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065236,"statfs":{"total":96636764160,"available":96322801664,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.443}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.46999999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:12.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:12.636 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:12.638 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:12.649 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-09-18T12:05:11.277397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16718,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094008,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021512,"statfs":{"total":289910292480,"available":288790028288,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":24644,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:09.054245+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:05:09.054245+0000","last_peered":"2024-09-18T12:05:09.054245+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:05:09.054245+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2561,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-991}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16717,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051719,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.498}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.44500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855481,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306604,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065236,"statfs":{"total":96636764160,"available":96322801664,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.443}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.46999999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:12.650 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:12.650 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19613, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156287+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:38.156287+0000', 'last_peered': '2024-09-18T12:04:38.156287+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:38.156287+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:12.651 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:12.651 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19750, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.160937+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:38.160937+0000', 'last_peered': '2024-09-18T12:04:38.160937+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:38.160937+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:12.651 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:12.651 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19510, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156169+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:38.156169+0000', 'last_peered': '2024-09-18T12:04:38.156169+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:38.156169+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:12.651 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:12.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19465, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157734+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:38.157734+0000', 'last_peered': '2024-09-18T12:04:38.157734+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:38.157734+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:12.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:05:12.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19463", 'reported_seq': 24644, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:09.054245+0000', 'last_change': '2024-09-18T12:04:38.127143+0000', 'last_active': '2024-09-18T12:05:09.054245+0000', 'last_peered': '2024-09-18T12:05:09.054245+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:05:09.054245+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19461, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2561, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -991}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:12.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:12.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26913, 'reported_epoch': 44, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:38.159248+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:38.159248+0000', 'last_peered': '2024-09-18T12:04:38.159248+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:38.159248+0000', 'last_undegraded': '2024-09-18T12:04:38.159248+0000', 'last_fullsized': '2024-09-18T12:04:38.159248+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:12.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:12.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19616, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156213+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:38.156213+0000', 'last_peered': '2024-09-18T12:04:38.156213+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:38.156213+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:12.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:05:12.652 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 19285, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157224+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:38.157224+0000', 'last_peered': '2024-09-18T12:04:38.157224+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:38.157224+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:12.653 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:12.653 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19603, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156505+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:38.156505+0000', 'last_peered': '2024-09-18T12:04:38.156505+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:38.156505+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:12.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:13.019 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:13.019 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25876/44 objects degraded (58809.091%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":713,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1120264192,"bytes_avail":288790028288,"bytes_total":289910292480,"degraded_objects":25876,"degraded_total":44,"degraded_ratio":588.09090909090912,"recovering_objects_per_sec":78,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":944,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:13.032 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:13.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:13.329 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:13.331 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:13.341 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-18T12:05:13.278011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16718,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098168,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017352,"statfs":{"total":289910292480,"available":288785768448,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-944,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003658"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":24644,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:09.054245+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:05:09.054245+0000","last_peered":"2024-09-18T12:05:09.054245+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:05:09.054245+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2561,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-991}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16717,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051719,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.498}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.44500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855482,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310764,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061076,"statfs":{"total":96636764160,"available":96318541824,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.45400000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.58099999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:16.344 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:16.677 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:16.687 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-09-18T12:05:15.278574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17280,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098168,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017352,"statfs":{"total":289910292480,"available":288785768448,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1506,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003565"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":25768,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:14.057255+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:05:14.057255+0000","last_peered":"2024-09-18T12:05:14.057255+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:05:14.057255+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-429}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17279,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051720,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.48399999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.32600000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855482,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310764,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061076,"statfs":{"total":96636764160,"available":96318541824,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.45400000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.58099999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.38700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:16.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:16.980 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:16.983 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:16.993 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-09-18T12:05:15.278574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17280,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098168,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017352,"statfs":{"total":289910292480,"available":288785768448,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1506,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003565"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":25768,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:14.057255+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:05:14.057255+0000","last_peered":"2024-09-18T12:05:14.057255+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:05:14.057255+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-429}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17279,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051720,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.48399999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.32600000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855482,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310764,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061076,"statfs":{"total":96636764160,"available":96318541824,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.45400000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.58099999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.38700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:16.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:16.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19613, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156287+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:04:38.156287+0000', 'last_peered': '2024-09-18T12:04:38.156287+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:04:38.156287+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:16.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:16.994 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19750, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.160937+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:04:38.160937+0000', 'last_peered': '2024-09-18T12:04:38.160937+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:04:38.160937+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:16.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:16.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19510, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156169+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:04:38.156169+0000', 'last_peered': '2024-09-18T12:04:38.156169+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:04:38.156169+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:16.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:16.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19465, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157734+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:04:38.157734+0000', 'last_peered': '2024-09-18T12:04:38.157734+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:04:38.157734+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:16.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-18T12:05:16.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "43'19463", 'reported_seq': 25768, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:14.057255+0000', 'last_change': '2024-09-18T12:04:38.127143+0000', 'last_active': '2024-09-18T12:05:14.057255+0000', 'last_peered': '2024-09-18T12:05:14.057255+0000', 'last_clean': '2024-09-18T11:55:02.235429+0000', 'last_became_active': '2024-09-18T11:55:04.722803+0000', 'last_became_peered': '2024-09-18T11:55:04.722803+0000', 'last_unstale': '2024-09-18T12:05:14.057255+0000', 'last_undegraded': '2024-09-18T11:55:04.291788+0000', 'last_fullsized': '2024-09-18T11:55:04.245769+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 431, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19461, 'num_write_kb': 38919, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3123, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -429}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:16.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:16.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26913, 'reported_epoch': 44, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:04:38.159248+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:04:38.159248+0000', 'last_peered': '2024-09-18T12:04:38.159248+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:04:38.159248+0000', 'last_undegraded': '2024-09-18T12:04:38.159248+0000', 'last_fullsized': '2024-09-18T12:04:38.159248+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:16.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:16.995 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19616, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156213+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:04:38.156213+0000', 'last_peered': '2024-09-18T12:04:38.156213+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:04:38.156213+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:16.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:05:16.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 19285, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.157224+0000', 'last_change': '2024-09-18T11:55:04.733107+0000', 'last_active': '2024-09-18T12:04:38.157224+0000', 'last_peered': '2024-09-18T12:04:38.157224+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:04:38.157224+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3595, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:16.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:16.996 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19603, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:04:38.156505+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:04:38.156505+0000', 'last_peered': '2024-09-18T12:04:38.156505+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:04:38.156505+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:16.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:17.373 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:17.373 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25314/44 objects degraded (57531.818%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":717,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1124524032,"bytes_avail":288785768448,"bytes_total":289910292480,"degraded_objects":25314,"degraded_total":44,"degraded_ratio":575.31818181818187,"recovering_objects_per_sec":125,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1506,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:17.391 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:17.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:17.719 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:17.721 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:17.736 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-18T12:05:17.489570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17280,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1085688,"kb_used_data":171344,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":282029832,"statfs":{"total":289910292480,"available":288798547968,"internally_reserved":0,"allocated":175456256,"data_stored":174645347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1059,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.213300"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19596,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160530+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:04:38.160530+0000","last_peered":"2024-09-18T12:04:38.160530+0000","last_clean":"2024-09-18T12:04:38.160530+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:04:38.160530+0000","last_undegraded":"2024-09-18T12:04:38.160530+0000","last_fullsized":"2024-09-18T12:04:38.160530+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19750,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.160937+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:04:38.160937+0000","last_peered":"2024-09-18T12:04:38.160937+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:04:38.160937+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19295,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160459+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:04:38.160459+0000","last_peered":"2024-09-18T12:04:38.160459+0000","last_clean":"2024-09-18T12:04:38.160459+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:04:38.160459+0000","last_undegraded":"2024-09-18T12:04:38.160459+0000","last_fullsized":"2024-09-18T12:04:38.160459+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.159308+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:04:38.159308+0000","last_peered":"2024-09-18T12:04:38.159308+0000","last_clean":"2024-09-18T12:04:38.159308+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:04:38.159308+0000","last_undegraded":"2024-09-18T12:04:38.159308+0000","last_fullsized":"2024-09-18T12:04:38.159308+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":25768,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:14.057255+0000","last_change":"2024-09-18T12:04:38.127143+0000","last_active":"2024-09-18T12:05:14.057255+0000","last_peered":"2024-09-18T12:05:14.057255+0000","last_clean":"2024-09-18T11:55:02.235429+0000","last_became_active":"2024-09-18T11:55:04.722803+0000","last_became_peered":"2024-09-18T11:55:04.722803+0000","last_unstale":"2024-09-18T12:05:14.057255+0000","last_undegraded":"2024-09-18T11:55:04.291788+0000","last_fullsized":"2024-09-18T11:55:04.245769+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":431,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-429}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26913,"reported_epoch":44,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:04:38.159248+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:04:38.159248+0000","last_peered":"2024-09-18T12:04:38.159248+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:04:38.159248+0000","last_undegraded":"2024-09-18T12:04:38.159248+0000","last_fullsized":"2024-09-18T12:04:38.159248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":45,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.160593+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:04:38.160593+0000","last_peered":"2024-09-18T12:04:38.160593+0000","last_clean":"2024-09-18T12:04:38.160593+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:04:38.160593+0000","last_undegraded":"2024-09-18T12:04:38.160593+0000","last_fullsized":"2024-09-18T12:04:38.160593+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17279,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051720,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420120,"kb_used_data":113928,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951720,"statfs":{"total":96636764160,"available":96206561280,"internally_reserved":0,"allocated":116662272,"data_stored":116407140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.48399999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:04:17 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.956,"15min":1.8660000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":16.847999999999999,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.782,"5min":1.4650000000000001,"15min":1.1299999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.0449999999999999,"5min":12.712,"15min":12.712},"last":0.32600000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855483,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298284,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073556,"statfs":{"total":96636764160,"available":96331321344,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.40799999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.38700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:20.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:21.097 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:21.099 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:21.109 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-09-18T12:05:19.279856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24883,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17711,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1085692,"kb_used_data":171352,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":282029828,"statfs":{"total":289910292480,"available":288798543872,"internally_reserved":0,"allocated":175464448,"data_stored":174648240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002459"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156287+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:04:38.156287+0000","last_peered":"2024-09-18T12:04:38.156287+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:04:38.156287+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19644,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155718+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:04:38.155718+0000","last_peered":"2024-09-18T12:04:38.155718+0000","last_clean":"2024-09-18T12:04:38.155718+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:04:38.155718+0000","last_undegraded":"2024-09-18T12:04:38.155718+0000","last_fullsized":"2024-09-18T12:04:38.155718+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19510,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156169+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:04:38.156169+0000","last_peered":"2024-09-18T12:04:38.156169+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:04:38.156169+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19465,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157734+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:04:38.157734+0000","last_peered":"2024-09-18T12:04:38.157734+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:04:38.157734+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155099+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:04:38.155099+0000","last_peered":"2024-09-18T12:04:38.155099+0000","last_clean":"2024-09-18T12:04:38.155099+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:04:38.155099+0000","last_undegraded":"2024-09-18T12:04:38.155099+0000","last_fullsized":"2024-09-18T12:04:38.155099+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26609,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T12:04:39.218592+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:04:39.218592+0000","last_peered":"2024-09-18T12:04:39.218592+0000","last_clean":"2024-09-18T12:04:39.218592+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:04:39.218592+0000","last_undegraded":"2024-09-18T12:04:39.218592+0000","last_fullsized":"2024-09-18T12:04:39.218592+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":97,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155118+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:04:38.155118+0000","last_peered":"2024-09-18T12:04:38.155118+0000","last_clean":"2024-09-18T12:04:38.155118+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:04:38.155118+0000","last_undegraded":"2024-09-18T12:04:38.155118+0000","last_fullsized":"2024-09-18T12:04:38.155118+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156213+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:04:38.156213+0000","last_peered":"2024-09-18T12:04:38.156213+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:04:38.156213+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.157224+0000","last_change":"2024-09-18T11:55:04.733107+0000","last_active":"2024-09-18T12:04:38.157224+0000","last_peered":"2024-09-18T12:04:38.157224+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:04:38.157224+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3591}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19603,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:04:38.156505+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:04:38.156505+0000","last_peered":"2024-09-18T12:04:38.156505+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:04:38.156505+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24883,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17710,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051721,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420124,"kb_used_data":113936,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951716,"statfs":{"total":96636764160,"available":96206557184,"internally_reserved":0,"allocated":116670464,"data_stored":116410033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38700000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.36199999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855483,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298284,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073556,"statfs":{"total":96636764160,"available":96331321344,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.40799999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367284,"kb_used_data":56996,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004556,"statfs":{"total":96636764160,"available":96260665344,"internally_reserved":0,"allocated":58363904,"data_stored":58105150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.38700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:21.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:21.399 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:21.401 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:21.413 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-18T12:05:21.280256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17971,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1085696,"kb_used_data":171360,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":282029824,"statfs":{"total":289910292480,"available":288798539776,"internally_reserved":0,"allocated":175472640,"data_stored":174651133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002245"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19808,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:19.382691+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:19.382691+0000","last_peered":"2024-09-18T12:05:19.382691+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:19.382691+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3333}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17970,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051721,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420124,"kb_used_data":113936,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951716,"statfs":{"total":96636764160,"available":96206557184,"internally_reserved":0,"allocated":116670464,"data_stored":116410033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38700000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.36199999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855483,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298284,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073556,"statfs":{"total":96636764160,"available":96331321344,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.40799999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247948,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367288,"kb_used_data":57004,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004552,"statfs":{"total":96636764160,"available":96260661248,"internally_reserved":0,"allocated":58372096,"data_stored":58108043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.433},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.56000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:21.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:21.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19615, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494308+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:17.494308+0000', 'last_peered': '2024-09-18T12:05:17.494308+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:17.494308+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:21.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:21.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19752, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.497182+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:05:17.497182+0000', 'last_peered': '2024-09-18T12:05:17.497182+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:05:17.497182+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:21.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:21.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19512, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494521+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:17.494521+0000', 'last_peered': '2024-09-18T12:05:17.494521+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:17.494521+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:21.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:21.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19467, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493192+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:17.493192+0000', 'last_peered': '2024-09-18T12:05:17.493192+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:17.493192+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:21.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:21.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26915, 'reported_epoch': 46, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:17.496045+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:17.496045+0000', 'last_peered': '2024-09-18T12:05:17.496045+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:17.496045+0000', 'last_undegraded': '2024-09-18T12:05:17.496045+0000', 'last_fullsized': '2024-09-18T12:05:17.496045+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:21.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:21.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19618, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493178+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:17.493178+0000', 'last_peered': '2024-09-18T12:05:17.493178+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:17.493178+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:21.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:05:21.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 19808, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:19.382691+0000', 'last_change': '2024-09-18T12:05:17.274231+0000', 'last_active': '2024-09-18T12:05:19.382691+0000', 'last_peered': '2024-09-18T12:05:19.382691+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:05:19.382691+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 262, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3333}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:21.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:21.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19605, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493137+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:17.493137+0000', 'last_peered': '2024-09-18T12:05:17.493137+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:17.493137+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:21.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:21.788 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:21.789 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24623/44 objects degraded (55961.364%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":721,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1111752704,"bytes_avail":288798539776,"bytes_total":289910292480,"degraded_objects":24623,"degraded_total":44,"degraded_ratio":559.61363636363637,"recovering_objects_per_sec":156,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:21.805 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:21.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:22.106 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:22.108 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:22.121 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-18T12:05:21.280256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17971,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1085696,"kb_used_data":171360,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":282029824,"statfs":{"total":289910292480,"available":288798539776,"internally_reserved":0,"allocated":175472640,"data_stored":174651133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002245"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.154916+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:04:38.154916+0000","last_peered":"2024-09-18T12:04:38.154916+0000","last_clean":"2024-09-18T12:04:38.154916+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:04:38.154916+0000","last_undegraded":"2024-09-18T12:04:38.154916+0000","last_fullsized":"2024-09-18T12:04:38.154916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26539,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155381+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:04:38.155381+0000","last_peered":"2024-09-18T12:04:38.155381+0000","last_clean":"2024-09-18T12:04:38.155381+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:04:38.155381+0000","last_undegraded":"2024-09-18T12:04:38.155381+0000","last_fullsized":"2024-09-18T12:04:38.155381+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":77,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.155072+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:04:38.155072+0000","last_peered":"2024-09-18T12:04:38.155072+0000","last_clean":"2024-09-18T12:04:38.155072+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:04:38.155072+0000","last_undegraded":"2024-09-18T12:04:38.155072+0000","last_fullsized":"2024-09-18T12:04:38.155072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157622+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:04:38.157622+0000","last_peered":"2024-09-18T12:04:38.157622+0000","last_clean":"2024-09-18T12:04:38.157622+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:04:38.157622+0000","last_undegraded":"2024-09-18T12:04:38.157622+0000","last_fullsized":"2024-09-18T12:04:38.157622+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157809+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:04:38.157809+0000","last_peered":"2024-09-18T12:04:38.157809+0000","last_clean":"2024-09-18T12:04:38.157809+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:04:38.157809+0000","last_undegraded":"2024-09-18T12:04:38.157809+0000","last_fullsized":"2024-09-18T12:04:38.157809+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19808,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:19.382691+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:19.382691+0000","last_peered":"2024-09-18T12:05:19.382691+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:19.382691+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3333}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157688+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:04:38.157688+0000","last_peered":"2024-09-18T12:04:38.157688+0000","last_clean":"2024-09-18T12:04:38.157688+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:04:38.157688+0000","last_undegraded":"2024-09-18T12:04:38.157688+0000","last_fullsized":"2024-09-18T12:04:38.157688+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T12:04:38.157766+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:04:38.157766+0000","last_peered":"2024-09-18T12:04:38.157766+0000","last_clean":"2024-09-18T12:04:38.157766+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:04:38.157766+0000","last_undegraded":"2024-09-18T12:04:38.157766+0000","last_fullsized":"2024-09-18T12:04:38.157766+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17970,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174317568,"data_stored":174268862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051721,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420124,"kb_used_data":113936,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951716,"statfs":{"total":96636764160,"available":96206557184,"internally_reserved":0,"allocated":116670464,"data_stored":116410033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38700000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.36199999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855483,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298284,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073556,"statfs":{"total":96636764160,"available":96331321344,"internally_reserved":0,"allocated":430080,"data_stored":133057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.40799999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247948,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367288,"kb_used_data":57004,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004552,"statfs":{"total":96636764160,"available":96260661248,"internally_reserved":0,"allocated":58372096,"data_stored":58108043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.433},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.56000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:25.123 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:25.448 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:25.462 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-18T12:05:25.281414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18641,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089888,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025632,"statfs":{"total":289910292480,"available":288794247168,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002170"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":21148,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:24.378123+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:24.378123+0000","last_peered":"2024-09-18T12:05:24.378123+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:24.378123+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18640,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051722,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.315},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.34499999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.36299999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855484,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.307},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.44500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:25.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:25.747 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:25.750 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:25.762 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-18T12:05:25.281414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18641,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089888,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025632,"statfs":{"total":289910292480,"available":288794247168,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002170"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":21148,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:24.378123+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:24.378123+0000","last_peered":"2024-09-18T12:05:24.378123+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:24.378123+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18640,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051722,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.315},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.34499999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.36299999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855484,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.307},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.44500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:25.764 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:25.764 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19615, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494308+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:17.494308+0000', 'last_peered': '2024-09-18T12:05:17.494308+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:17.494308+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:25.764 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:25.764 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19752, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.497182+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:05:17.497182+0000', 'last_peered': '2024-09-18T12:05:17.497182+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:05:17.497182+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:25.764 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:25.764 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19512, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494521+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:17.494521+0000', 'last_peered': '2024-09-18T12:05:17.494521+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:17.494521+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:25.764 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:25.764 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19467, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493192+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:17.493192+0000', 'last_peered': '2024-09-18T12:05:17.493192+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:17.493192+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:25.765 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:25.765 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26915, 'reported_epoch': 46, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:17.496045+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:17.496045+0000', 'last_peered': '2024-09-18T12:05:17.496045+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:17.496045+0000', 'last_undegraded': '2024-09-18T12:05:17.496045+0000', 'last_fullsized': '2024-09-18T12:05:17.496045+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:25.765 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:25.765 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19618, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493178+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:17.493178+0000', 'last_peered': '2024-09-18T12:05:17.493178+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:17.493178+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:25.765 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:05:25.765 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 21148, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:24.378123+0000', 'last_change': '2024-09-18T12:05:17.274231+0000', 'last_active': '2024-09-18T12:05:24.378123+0000', 'last_peered': '2024-09-18T12:05:24.378123+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:05:24.378123+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2665, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 932, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2663}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:25.765 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:25.765 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19605, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493137+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:17.493137+0000', 'last_peered': '2024-09-18T12:05:17.493137+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:17.493137+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:25.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:26.146 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:26.147 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23953/44 objects degraded (54438.636%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":726,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1116045312,"bytes_avail":288794247168,"bytes_total":289910292480,"degraded_objects":23953,"degraded_total":44,"degraded_ratio":544.38636363636363,"recovering_objects_per_sec":170,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:26.160 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:26.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:26.457 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:26.460 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:26.473 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-18T12:05:25.281414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18641,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089888,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025632,"statfs":{"total":289910292480,"available":288794247168,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002170"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":21148,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:24.378123+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:24.378123+0000","last_peered":"2024-09-18T12:05:24.378123+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:24.378123+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18640,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051722,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.315},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.34499999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.36299999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855484,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":3.0859999999999999,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.307},{"interface":"front","average":{"1min":1.155,"5min":1.466,"15min":1.2829999999999999},"min":{"1min":0.193,"5min":0.185,"15min":0.185},"max":{"1min":8.3200000000000003,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:04:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9690000000000001,"5min":1.927,"15min":1.3839999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":11.41,"5min":13.737,"15min":13.737},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.85699999999999998,"15min":0.78400000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":4.3479999999999999,"5min":8.4730000000000008,"15min":11.996},"last":0.44500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:29.475 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:29.937 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:29.939 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:29.951 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-18T12:05:29.282587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18641,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098208,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017312,"statfs":{"total":289910292480,"available":288785727488,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.789200"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":21148,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:24.378123+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:24.378123+0000","last_peered":"2024-09-18T12:05:24.378123+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:24.378123+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18640,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051723,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.44600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.5},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.47999999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855485,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310772,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061068,"statfs":{"total":96636764160,"available":96318533632,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.26000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.33800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:29.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:30.333 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:30.335 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:30.346 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-18T12:05:29.282587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18641,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098208,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017312,"statfs":{"total":289910292480,"available":288785727488,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.789200"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":21148,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:24.378123+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:24.378123+0000","last_peered":"2024-09-18T12:05:24.378123+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:24.378123+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18640,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051723,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.44600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.5},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.47999999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855485,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310772,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061068,"statfs":{"total":96636764160,"available":96318533632,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.26000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.33800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:30.348 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:30.348 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19615, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494308+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:17.494308+0000', 'last_peered': '2024-09-18T12:05:17.494308+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:17.494308+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:30.348 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:30.348 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19752, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.497182+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:05:17.497182+0000', 'last_peered': '2024-09-18T12:05:17.497182+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:05:17.497182+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:30.348 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:30.348 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19512, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494521+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:17.494521+0000', 'last_peered': '2024-09-18T12:05:17.494521+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:17.494521+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:30.348 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:30.348 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19467, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493192+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:17.493192+0000', 'last_peered': '2024-09-18T12:05:17.493192+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:17.493192+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:30.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:30.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26915, 'reported_epoch': 46, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:17.496045+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:17.496045+0000', 'last_peered': '2024-09-18T12:05:17.496045+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:17.496045+0000', 'last_undegraded': '2024-09-18T12:05:17.496045+0000', 'last_fullsized': '2024-09-18T12:05:17.496045+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:30.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:30.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19618, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493178+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:17.493178+0000', 'last_peered': '2024-09-18T12:05:17.493178+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:17.493178+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:30.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:05:30.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 21148, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:24.378123+0000', 'last_change': '2024-09-18T12:05:17.274231+0000', 'last_active': '2024-09-18T12:05:24.378123+0000', 'last_peered': '2024-09-18T12:05:24.378123+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:05:24.378123+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2665, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 932, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2663}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:30.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:30.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19605, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493137+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:17.493137+0000', 'last_peered': '2024-09-18T12:05:17.493137+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:17.493137+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:30.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:30.734 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:30.734 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23953/44 objects degraded (54438.636%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":730,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1124564992,"bytes_avail":288785727488,"bytes_total":289910292480,"degraded_objects":23953,"degraded_total":44,"degraded_ratio":544.38636363636363,"recovering_objects_per_sec":126,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:30.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:30.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:31.154 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:31.156 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:31.168 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-18T12:05:29.282587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18641,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098208,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017312,"statfs":{"total":289910292480,"available":288785727488,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.789200"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":21148,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:24.378123+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:24.378123+0000","last_peered":"2024-09-18T12:05:24.378123+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:24.378123+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2665,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18640,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051723,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.44600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.5},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.47999999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855485,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310772,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061068,"statfs":{"total":96636764160,"available":96318533632,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.26000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.33800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1990000000000001,"15min":1.073},"min":{"1min":0.20899999999999999,"5min":0.161,"15min":0.161},"max":{"1min":5.1749999999999998,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.88200000000000001,"15min":1.004},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":3.2440000000000002,"5min":8.3460000000000001,"15min":8.3460000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:34.170 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:34.497 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:34.499 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:34.510 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-18T12:05:33.283698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19342,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1085728,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":282029792,"statfs":{"total":289910292480,"available":288798507008,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1371,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003442"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":22550,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:29.378826+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:29.378826+0000","last_peered":"2024-09-18T12:05:29.378826+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:29.378826+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19341,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051723,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.44600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.5},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.47999999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855486,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298292,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073548,"statfs":{"total":96636764160,"available":96331313152,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.183},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.32700000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.32000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.38900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:34.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:34.816 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:34.818 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:34.829 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-18T12:05:33.283698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19342,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1085728,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":282029792,"statfs":{"total":289910292480,"available":288798507008,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1371,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003442"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":22550,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:29.378826+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:29.378826+0000","last_peered":"2024-09-18T12:05:29.378826+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:29.378826+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19341,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051723,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.44600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.5},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.47999999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855486,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298292,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073548,"statfs":{"total":96636764160,"available":96331313152,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.183},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.32700000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.32000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.38900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:34.830 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:34.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19615, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494308+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:17.494308+0000', 'last_peered': '2024-09-18T12:05:17.494308+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:17.494308+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:34.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:34.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19752, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.497182+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:05:17.497182+0000', 'last_peered': '2024-09-18T12:05:17.497182+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:05:17.497182+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:34.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:34.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19512, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494521+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:17.494521+0000', 'last_peered': '2024-09-18T12:05:17.494521+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:17.494521+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:34.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:34.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19467, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493192+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:17.493192+0000', 'last_peered': '2024-09-18T12:05:17.493192+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:17.493192+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:34.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:34.831 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26915, 'reported_epoch': 46, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:17.496045+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:17.496045+0000', 'last_peered': '2024-09-18T12:05:17.496045+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:17.496045+0000', 'last_undegraded': '2024-09-18T12:05:17.496045+0000', 'last_fullsized': '2024-09-18T12:05:17.496045+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:34.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:34.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19618, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493178+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:17.493178+0000', 'last_peered': '2024-09-18T12:05:17.493178+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:17.493178+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:34.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:05:34.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 22550, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:29.378826+0000', 'last_change': '2024-09-18T12:05:17.274231+0000', 'last_active': '2024-09-18T12:05:29.378826+0000', 'last_peered': '2024-09-18T12:05:29.378826+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:05:29.378826+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1964, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1633, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1962}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:34.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:34.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19605, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493137+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:17.493137+0000', 'last_peered': '2024-09-18T12:05:17.493137+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:17.493137+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:34.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:35.171 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:35.171 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23252/44 objects degraded (52845.455%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":735,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1111785472,"bytes_avail":288798507008,"bytes_total":289910292480,"degraded_objects":23252,"degraded_total":44,"degraded_ratio":528.4545454545455,"recovering_objects_per_sec":114,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1371,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:35.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:35.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:35.679 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:35.682 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:35.692 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-09-18T12:05:35.284370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22700,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19894,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1085728,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":282029792,"statfs":{"total":289910292480,"available":288798507008,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1923,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003486"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":23654,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:34.384596+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:34.384596+0000","last_peered":"2024-09-18T12:05:34.384596+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:34.384596+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22700,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19893,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051724,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.432}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.45900000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855486,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298292,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073548,"statfs":{"total":96636764160,"available":96331313152,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.183},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.23100000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.32700000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.44400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:38.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:39.026 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:39.028 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:39.039 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-18T12:05:37.285130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22700,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19894,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089888,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025632,"statfs":{"total":289910292480,"available":288794247168,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003715"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":23654,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:34.384596+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:34.384596+0000","last_peered":"2024-09-18T12:05:34.384596+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:34.384596+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22700,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19893,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051724,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:04:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.0669999999999999,"15min":0.70999999999999996},"min":{"1min":0.19700000000000001,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":5.6840000000000002,"5min":19.776,"15min":19.776},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.472,"5min":1.901,"15min":1.1339999999999999},"min":{"1min":0.187,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":5.8029999999999999,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.432}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.45900000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855487,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.249},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.35199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.44400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:39.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:39.325 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:39.327 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:39.337 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-18T12:05:39.285794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22700,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19894,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089888,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025632,"statfs":{"total":289910292480,"available":288794247168,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003846"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":23654,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:34.384596+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:34.384596+0000","last_peered":"2024-09-18T12:05:34.384596+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:34.384596+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22700,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19893,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051725,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.495},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.56699999999999995}]}]},{"osd":1,"up_from":35,"seq":150323855487,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.249},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.35199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.44400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:39.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:39.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19615, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494308+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:17.494308+0000', 'last_peered': '2024-09-18T12:05:17.494308+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:17.494308+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:39.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:39.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19752, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.497182+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:05:17.497182+0000', 'last_peered': '2024-09-18T12:05:17.497182+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:05:17.497182+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:39.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:39.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19512, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494521+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:17.494521+0000', 'last_peered': '2024-09-18T12:05:17.494521+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:17.494521+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:39.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:39.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19467, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493192+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:17.493192+0000', 'last_peered': '2024-09-18T12:05:17.493192+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:17.493192+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:39.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:39.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26915, 'reported_epoch': 46, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:17.496045+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:17.496045+0000', 'last_peered': '2024-09-18T12:05:17.496045+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:17.496045+0000', 'last_undegraded': '2024-09-18T12:05:17.496045+0000', 'last_fullsized': '2024-09-18T12:05:17.496045+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:39.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:39.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19618, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493178+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:17.493178+0000', 'last_peered': '2024-09-18T12:05:17.493178+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:17.493178+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:39.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:05:39.340 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 23654, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:34.384596+0000', 'last_change': '2024-09-18T12:05:17.274231+0000', 'last_active': '2024-09-18T12:05:34.384596+0000', 'last_peered': '2024-09-18T12:05:34.384596+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:05:34.384596+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1412, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2185, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1410}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:39.340 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:39.340 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19605, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493137+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:17.493137+0000', 'last_peered': '2024-09-18T12:05:17.493137+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:17.493137+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:39.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:39.692 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:39.692 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22700/44 objects degraded (51590.909%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":739,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1116045312,"bytes_avail":288794247168,"bytes_total":289910292480,"degraded_objects":22700,"degraded_total":44,"degraded_ratio":515.90909090909088,"recovering_objects_per_sec":104,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:39.704 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:39.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:39.991 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:39.994 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:40.005 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-18T12:05:39.285794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22700,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19894,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089888,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025632,"statfs":{"total":289910292480,"available":288794247168,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003846"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":23654,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:34.384596+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:34.384596+0000","last_peered":"2024-09-18T12:05:34.384596+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:34.384596+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22700,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19893,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051725,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.495},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.56699999999999995}]}]},{"osd":1,"up_from":35,"seq":150323855487,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.249},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.35199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:04:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.778,"15min":1.1379999999999999},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":8.8650000000000002,"5min":13.773,"15min":13.773},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.452,"15min":0.94799999999999995},"min":{"1min":0.245,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":8.6430000000000007,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.44400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:43.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:43.319 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:43.322 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:43.333 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-18T12:05:43.287524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20397,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094048,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021472,"statfs":{"total":289910292480,"available":288789987328,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004547"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":24660,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:39.380757+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:39.380757+0000","last_peered":"2024-09-18T12:05:39.380757+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:39.380757+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2688,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20396,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051725,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.495},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.56699999999999995}]}]},{"osd":1,"up_from":35,"seq":150323855488,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306612,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065228,"statfs":{"total":96636764160,"available":96322793472,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.497},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.35299999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.58699999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.38500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.54600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:43.335 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:43.626 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:43.629 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:43.642 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-18T12:05:43.287524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20397,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094048,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021472,"statfs":{"total":289910292480,"available":288789987328,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004547"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":24660,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:39.380757+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:39.380757+0000","last_peered":"2024-09-18T12:05:39.380757+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:39.380757+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2688,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20396,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051725,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.495},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.56699999999999995}]}]},{"osd":1,"up_from":35,"seq":150323855488,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306612,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065228,"statfs":{"total":96636764160,"available":96322793472,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.497},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.35299999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.58699999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.38500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.54600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:43.644 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:43.644 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19615, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494308+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:17.494308+0000', 'last_peered': '2024-09-18T12:05:17.494308+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:17.494308+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:43.644 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:43.644 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19752, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.497182+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:05:17.497182+0000', 'last_peered': '2024-09-18T12:05:17.497182+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:05:17.497182+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:43.644 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:43.644 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19512, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494521+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:17.494521+0000', 'last_peered': '2024-09-18T12:05:17.494521+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:17.494521+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:43.644 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:43.644 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19467, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493192+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:17.493192+0000', 'last_peered': '2024-09-18T12:05:17.493192+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:17.493192+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:43.645 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:43.645 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26915, 'reported_epoch': 46, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:17.496045+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:17.496045+0000', 'last_peered': '2024-09-18T12:05:17.496045+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:17.496045+0000', 'last_undegraded': '2024-09-18T12:05:17.496045+0000', 'last_fullsized': '2024-09-18T12:05:17.496045+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:43.645 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:43.645 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19618, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493178+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:17.493178+0000', 'last_peered': '2024-09-18T12:05:17.493178+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:17.493178+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:43.645 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:05:43.645 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 24660, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:39.380757+0000', 'last_change': '2024-09-18T12:05:17.274231+0000', 'last_active': '2024-09-18T12:05:39.380757+0000', 'last_peered': '2024-09-18T12:05:39.380757+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:05:39.380757+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2688, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -907}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:43.645 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:43.645 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19605, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493137+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:17.493137+0000', 'last_peered': '2024-09-18T12:05:17.493137+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:17.493137+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:43.645 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:44.015 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:44.016 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22197/44 objects degraded (50447.727%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":743,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":47,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1120305152,"bytes_avail":288789987328,"bytes_total":289910292480,"degraded_objects":22197,"degraded_total":44,"degraded_ratio":504.47727272727275,"recovering_objects_per_sec":87,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1055,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:44.032 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:44.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:44.317 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:44.327 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-18T12:05:43.287524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20397,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094048,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021472,"statfs":{"total":289910292480,"available":288789987328,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004547"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":24660,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:39.380757+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:39.380757+0000","last_peered":"2024-09-18T12:05:39.380757+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:39.380757+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2688,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20396,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051725,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.495},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.56699999999999995}]}]},{"osd":1,"up_from":35,"seq":150323855488,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306612,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065228,"statfs":{"total":96636764160,"available":96322793472,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.497},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.35299999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.58699999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.38500000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.54600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:47.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:47.692 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:47.703 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-09-18T12:05:47.656641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20977,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098208,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017312,"statfs":{"total":289910292480,"available":288785727488,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1083,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.371511"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":25820,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:44.387578+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:44.387578+0000","last_peered":"2024-09-18T12:05:44.387578+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:44.387578+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3268,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20976,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051726,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33100000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.35399999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855489,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310772,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061068,"statfs":{"total":96636764160,"available":96318533632,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.314},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.40799999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.58599999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.501}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:47.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:48.014 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:48.017 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:48.026 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-09-18T12:05:47.656641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20977,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098208,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017312,"statfs":{"total":289910292480,"available":288785727488,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1083,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.371511"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":25820,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:44.387578+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:44.387578+0000","last_peered":"2024-09-18T12:05:44.387578+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:44.387578+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3268,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20976,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051726,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33100000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.35399999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855489,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310772,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061068,"statfs":{"total":96636764160,"available":96318533632,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.314},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.40799999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.58599999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.501}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:48.028 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:48.028 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19615, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494308+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:17.494308+0000', 'last_peered': '2024-09-18T12:05:17.494308+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:17.494308+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:48.028 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:48.029 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19752, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.497182+0000', 'last_change': '2024-09-18T11:55:04.787996+0000', 'last_active': '2024-09-18T12:05:17.497182+0000', 'last_peered': '2024-09-18T12:05:17.497182+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:05:17.497182+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:48.029 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:48.029 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19512, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.494521+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:17.494521+0000', 'last_peered': '2024-09-18T12:05:17.494521+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:17.494521+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:48.029 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:48.029 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19467, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493192+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:17.493192+0000', 'last_peered': '2024-09-18T12:05:17.493192+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:17.493192+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:48.029 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:48.029 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26915, 'reported_epoch': 46, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:17.496045+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:17.496045+0000', 'last_peered': '2024-09-18T12:05:17.496045+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:17.496045+0000', 'last_undegraded': '2024-09-18T12:05:17.496045+0000', 'last_fullsized': '2024-09-18T12:05:17.496045+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:48.029 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:48.029 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19618, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493178+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:17.493178+0000', 'last_peered': '2024-09-18T12:05:17.493178+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:17.493178+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:48.030 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:05:48.030 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 25820, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:44.387578+0000', 'last_change': '2024-09-18T12:05:17.274231+0000', 'last_active': '2024-09-18T12:05:44.387578+0000', 'last_peered': '2024-09-18T12:05:44.387578+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:05:44.387578+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 329, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3268, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -327}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:48.030 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:48.030 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19605, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:17.493137+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:17.493137+0000', 'last_peered': '2024-09-18T12:05:17.493137+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:17.493137+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:48.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:48.420 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:48.420 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21617/44 objects degraded (49129.545%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":748,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1124564992,"bytes_avail":288785727488,"bytes_total":289910292480,"degraded_objects":21617,"degraded_total":44,"degraded_ratio":491.29545454545456,"recovering_objects_per_sec":90,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1083,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:48.434 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:48.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:48.740 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:48.743 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:48.752 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-18T12:05:48.660009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20977,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098208,"kb_used_data":171384,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017312,"statfs":{"total":289910292480,"available":288785727488,"internally_reserved":0,"allocated":175497216,"data_stored":174661408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1083,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.374215"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494308+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:17.494308+0000","last_peered":"2024-09-18T12:05:17.494308+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:17.494308+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19598,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497194+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:17.497194+0000","last_peered":"2024-09-18T12:05:17.497194+0000","last_clean":"2024-09-18T12:05:17.497194+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:17.497194+0000","last_undegraded":"2024-09-18T12:05:17.497194+0000","last_fullsized":"2024-09-18T12:05:17.497194+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19752,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.497182+0000","last_change":"2024-09-18T11:55:04.787996+0000","last_active":"2024-09-18T12:05:17.497182+0000","last_peered":"2024-09-18T12:05:17.497182+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:17.497182+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3536}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19297,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496817+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:17.496817+0000","last_peered":"2024-09-18T12:05:17.496817+0000","last_clean":"2024-09-18T12:05:17.496817+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:17.496817+0000","last_undegraded":"2024-09-18T12:05:17.496817+0000","last_fullsized":"2024-09-18T12:05:17.496817+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19646,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491954+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:17.491954+0000","last_peered":"2024-09-18T12:05:17.491954+0000","last_clean":"2024-09-18T12:05:17.491954+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:17.491954+0000","last_undegraded":"2024-09-18T12:05:17.491954+0000","last_fullsized":"2024-09-18T12:05:17.491954+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19512,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.494521+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:17.494521+0000","last_peered":"2024-09-18T12:05:17.494521+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:17.494521+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19467,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493192+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:17.493192+0000","last_peered":"2024-09-18T12:05:17.493192+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:17.493192+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19835,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491237+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:17.491237+0000","last_peered":"2024-09-18T12:05:17.491237+0000","last_clean":"2024-09-18T12:05:17.491237+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:17.491237+0000","last_undegraded":"2024-09-18T12:05:17.491237+0000","last_fullsized":"2024-09-18T12:05:17.491237+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.502571+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:17.502571+0000","last_peered":"2024-09-18T12:05:17.502571+0000","last_clean":"2024-09-18T12:05:17.502571+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:17.502571+0000","last_undegraded":"2024-09-18T12:05:17.502571+0000","last_fullsized":"2024-09-18T12:05:17.502571+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26611,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491801+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:17.491801+0000","last_peered":"2024-09-18T12:05:17.491801+0000","last_clean":"2024-09-18T12:05:17.491801+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:17.491801+0000","last_undegraded":"2024-09-18T12:05:17.491801+0000","last_fullsized":"2024-09-18T12:05:17.491801+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":99,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.491617+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:17.491617+0000","last_peered":"2024-09-18T12:05:17.491617+0000","last_clean":"2024-09-18T12:05:17.491617+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:17.491617+0000","last_undegraded":"2024-09-18T12:05:17.491617+0000","last_fullsized":"2024-09-18T12:05:17.491617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26642,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T12:05:18.557713+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:18.557713+0000","last_peered":"2024-09-18T12:05:18.557713+0000","last_clean":"2024-09-18T12:05:18.557713+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:18.557713+0000","last_undegraded":"2024-09-18T12:05:18.557713+0000","last_fullsized":"2024-09-18T12:05:18.557713+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26915,"reported_epoch":46,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:17.496045+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:17.496045+0000","last_peered":"2024-09-18T12:05:17.496045+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:17.496045+0000","last_undegraded":"2024-09-18T12:05:17.496045+0000","last_fullsized":"2024-09-18T12:05:17.496045+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493178+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:17.493178+0000","last_peered":"2024-09-18T12:05:17.493178+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:17.493178+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":47,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.498298+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:17.498298+0000","last_peered":"2024-09-18T12:05:17.498298+0000","last_clean":"2024-09-18T12:05:17.498298+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:17.498298+0000","last_undegraded":"2024-09-18T12:05:17.498298+0000","last_fullsized":"2024-09-18T12:05:17.498298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":25820,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:44.387578+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:44.387578+0000","last_peered":"2024-09-18T12:05:44.387578+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:44.387578+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3268,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19605,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:17.493137+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:17.493137+0000","last_peered":"2024-09-18T12:05:17.493137+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:17.493137+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20976,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051726,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420136,"kb_used_data":113944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951704,"statfs":{"total":96636764160,"available":96206544896,"internally_reserved":0,"allocated":116678656,"data_stored":116412905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33100000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.35399999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855489,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310772,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061068,"statfs":{"total":96636764160,"available":96318533632,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.314},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.40799999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.58599999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367300,"kb_used_data":57012,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004540,"statfs":{"total":96636764160,"available":96260648960,"internally_reserved":0,"allocated":58380288,"data_stored":58110915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.501}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:51.755 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:52.098 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:52.100 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:52.111 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-18T12:05:51.290360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21179,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098216,"kb_used_data":171400,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017304,"statfs":{"total":289910292480,"available":288785719296,"internally_reserved":0,"allocated":175513600,"data_stored":174667144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":782,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002836"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":20159,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:49.063883+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:05:49.063883+0000","last_peered":"2024-09-18T12:05:49.063883+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:49.063883+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":25820,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:44.387578+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:44.387578+0000","last_peered":"2024-09-18T12:05:44.387578+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:44.387578+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3268,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21178,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051727,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420140,"kb_used_data":113952,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951700,"statfs":{"total":96636764160,"available":96206540800,"internally_reserved":0,"allocated":116686848,"data_stored":116415773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33100000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.35399999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855489,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310772,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061068,"statfs":{"total":96636764160,"available":96318533632,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.314},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.40799999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.58599999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367304,"kb_used_data":57020,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004536,"statfs":{"total":96636764160,"available":96260644864,"internally_reserved":0,"allocated":58388480,"data_stored":58113783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.314},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.27900000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:52.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:52.400 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:52.403 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:52.413 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-18T12:05:51.290360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21179,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098216,"kb_used_data":171400,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017304,"statfs":{"total":289910292480,"available":288785719296,"internally_reserved":0,"allocated":175513600,"data_stored":174667144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":782,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002836"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":20159,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:49.063883+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:05:49.063883+0000","last_peered":"2024-09-18T12:05:49.063883+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:49.063883+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":25820,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:44.387578+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:44.387578+0000","last_peered":"2024-09-18T12:05:44.387578+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:44.387578+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3268,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21178,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051727,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420140,"kb_used_data":113952,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951700,"statfs":{"total":96636764160,"available":96206540800,"internally_reserved":0,"allocated":116686848,"data_stored":116415773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33100000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.35399999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855489,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310772,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061068,"statfs":{"total":96636764160,"available":96318533632,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.314},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.40799999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.58599999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367304,"kb_used_data":57020,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004536,"statfs":{"total":96636764160,"available":96260644864,"internally_reserved":0,"allocated":58388480,"data_stored":58113783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.314},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.27900000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:52.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:52.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19617, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.664580+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:47.664580+0000', 'last_peered': '2024-09-18T12:05:47.664580+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:47.664580+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:52.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:52.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 20159, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:49.063883+0000', 'last_change': '2024-09-18T12:05:46.861100+0000', 'last_active': '2024-09-18T12:05:49.063883+0000', 'last_peered': '2024-09-18T12:05:49.063883+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:05:49.063883+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 202, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:52.414 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:52.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19514, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662385+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:47.662385+0000', 'last_peered': '2024-09-18T12:05:47.662385+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:47.662385+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:52.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:52.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19469, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663597+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:47.663597+0000', 'last_peered': '2024-09-18T12:05:47.663597+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:47.663597+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:52.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:52.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26917, 'reported_epoch': 48, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:47.657898+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:47.657898+0000', 'last_peered': '2024-09-18T12:05:47.657898+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:47.657898+0000', 'last_undegraded': '2024-09-18T12:05:47.657898+0000', 'last_fullsized': '2024-09-18T12:05:47.657898+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:52.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:52.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19620, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663179+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:47.663179+0000', 'last_peered': '2024-09-18T12:05:47.663179+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:47.663179+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:52.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-18T12:05:52.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "43'19198", 'reported_seq': 25820, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:44.387578+0000', 'last_change': '2024-09-18T12:05:17.274231+0000', 'last_active': '2024-09-18T12:05:44.387578+0000', 'last_peered': '2024-09-18T12:05:44.387578+0000', 'last_clean': '2024-09-18T11:55:02.215803+0000', 'last_became_active': '2024-09-18T11:55:04.724471+0000', 'last_became_peered': '2024-09-18T11:55:04.724471+0000', 'last_unstale': '2024-09-18T12:05:44.387578+0000', 'last_undegraded': '2024-09-18T11:55:04.323648+0000', 'last_fullsized': '2024-09-18T11:55:04.249130+0000', 'mapping_epoch': 36, 'log_start': "39'9100", 'ondisk_log_start': "39'9100", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 329, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19196, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3268, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -327}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:52.415 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:52.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19607, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662016+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:47.662016+0000', 'last_peered': '2024-09-18T12:05:47.662016+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:47.662016+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:52.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:52.770 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:52.770 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21415/44 objects degraded (48670.455%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":752,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":2},{"state_name":"active+recovery_wait","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1124573184,"bytes_avail":288785719296,"bytes_total":289910292480,"degraded_objects":21415,"degraded_total":44,"degraded_ratio":486.70454545454544,"recovering_objects_per_sec":97,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":782,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:52.783 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:52.783 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:53.069 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:53.071 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:53.081 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-18T12:05:51.290360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21179,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1098216,"kb_used_data":171400,"kb_used_omap":13,"kb_used_meta":926770,"kb_avail":282017304,"statfs":{"total":289910292480,"available":288785719296,"internally_reserved":0,"allocated":175513600,"data_stored":174667144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":949013010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":782,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002836"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":20159,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:49.063883+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:05:49.063883+0000","last_peered":"2024-09-18T12:05:49.063883+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:49.063883+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496359+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:17.496359+0000","last_peered":"2024-09-18T12:05:17.496359+0000","last_clean":"2024-09-18T12:05:17.496359+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:17.496359+0000","last_undegraded":"2024-09-18T12:05:17.496359+0000","last_fullsized":"2024-09-18T12:05:17.496359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26541,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496589+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:17.496589+0000","last_peered":"2024-09-18T12:05:17.496589+0000","last_clean":"2024-09-18T12:05:17.496589+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:17.496589+0000","last_undegraded":"2024-09-18T12:05:17.496589+0000","last_fullsized":"2024-09-18T12:05:17.496589+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496415+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:17.496415+0000","last_peered":"2024-09-18T12:05:17.496415+0000","last_clean":"2024-09-18T12:05:17.496415+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:17.496415+0000","last_undegraded":"2024-09-18T12:05:17.496415+0000","last_fullsized":"2024-09-18T12:05:17.496415+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496889+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:17.496889+0000","last_peered":"2024-09-18T12:05:17.496889+0000","last_clean":"2024-09-18T12:05:17.496889+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:17.496889+0000","last_undegraded":"2024-09-18T12:05:17.496889+0000","last_fullsized":"2024-09-18T12:05:17.496889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26676,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497006+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:17.497006+0000","last_peered":"2024-09-18T12:05:17.497006+0000","last_clean":"2024-09-18T12:05:17.497006+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:17.497006+0000","last_undegraded":"2024-09-18T12:05:17.497006+0000","last_fullsized":"2024-09-18T12:05:17.497006+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":25820,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:44.387578+0000","last_change":"2024-09-18T12:05:17.274231+0000","last_active":"2024-09-18T12:05:44.387578+0000","last_peered":"2024-09-18T12:05:44.387578+0000","last_clean":"2024-09-18T11:55:02.215803+0000","last_became_active":"2024-09-18T11:55:04.724471+0000","last_became_peered":"2024-09-18T11:55:04.724471+0000","last_unstale":"2024-09-18T12:05:44.387578+0000","last_undegraded":"2024-09-18T11:55:04.323648+0000","last_fullsized":"2024-09-18T11:55:04.249130+0000","mapping_epoch":36,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3268,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.496870+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:17.496870+0000","last_peered":"2024-09-18T12:05:17.496870+0000","last_clean":"2024-09-18T12:05:17.496870+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:17.496870+0000","last_undegraded":"2024-09-18T12:05:17.496870+0000","last_fullsized":"2024-09-18T12:05:17.496870+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T12:05:17.497072+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:17.497072+0000","last_peered":"2024-09-18T12:05:17.497072+0000","last_clean":"2024-09-18T12:05:17.497072+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:17.497072+0000","last_undegraded":"2024-09-18T12:05:17.497072+0000","last_fullsized":"2024-09-18T12:05:17.497072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21178,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174309376,"data_stored":174267628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051727,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420140,"kb_used_data":113952,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951700,"statfs":{"total":96636764160,"available":96206540800,"internally_reserved":0,"allocated":116686848,"data_stored":116415773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33100000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.35399999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855489,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310772,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94061068,"statfs":{"total":96636764160,"available":96318533632,"internally_reserved":0,"allocated":438272,"data_stored":137588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.314},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.40799999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.58599999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367304,"kb_used_data":57020,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004536,"statfs":{"total":96636764160,"available":96260644864,"internally_reserved":0,"allocated":58388480,"data_stored":58113783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.314},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.27900000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":6170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:56.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:56.413 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:56.416 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:56.426 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-18T12:05:55.291502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21859,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1085772,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":282029748,"statfs":{"total":289910292480,"available":288798461952,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002505"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":20861,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:54.056089+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:05:54.056089+0000","last_peered":"2024-09-18T12:05:54.056089+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:54.056089+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2984}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21858,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051728,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.42099999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.46100000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855490,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298304,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073536,"statfs":{"total":96636764160,"available":96331300864,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.46999999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.36199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.436},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.502},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.48999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:56.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:56.730 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:56.732 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:56.747 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-18T12:05:55.291502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21859,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1085772,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":914290,"kb_avail":282029748,"statfs":{"total":289910292480,"available":288798461952,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":936233490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002505"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":20861,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:54.056089+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:05:54.056089+0000","last_peered":"2024-09-18T12:05:54.056089+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:54.056089+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2984}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21858,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051728,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.42099999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.46100000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855490,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298304,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073536,"statfs":{"total":96636764160,"available":96331300864,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":304999942},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.46999999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.36199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.436},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.502},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.48999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:05:56.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:05:56.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19617, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.664580+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:47.664580+0000', 'last_peered': '2024-09-18T12:05:47.664580+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:47.664580+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:56.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:05:56.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 20861, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:54.056089+0000', 'last_change': '2024-09-18T12:05:46.861100+0000', 'last_active': '2024-09-18T12:05:54.056089+0000', 'last_peered': '2024-09-18T12:05:54.056089+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:05:54.056089+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2984, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 553, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2984}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:56.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:05:56.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19514, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662385+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:47.662385+0000', 'last_peered': '2024-09-18T12:05:47.662385+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:47.662385+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:56.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:05:56.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19469, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663597+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:47.663597+0000', 'last_peered': '2024-09-18T12:05:47.663597+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:47.663597+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:56.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:05:56.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26917, 'reported_epoch': 48, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:47.657898+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:47.657898+0000', 'last_peered': '2024-09-18T12:05:47.657898+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:47.657898+0000', 'last_undegraded': '2024-09-18T12:05:47.657898+0000', 'last_fullsized': '2024-09-18T12:05:47.657898+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:05:56.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:05:56.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19620, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663179+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:47.663179+0000', 'last_peered': '2024-09-18T12:05:47.663179+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:47.663179+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:56.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:05:56.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19607, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662016+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:47.662016+0000', 'last_peered': '2024-09-18T12:05:47.662016+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:47.662016+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:05:56.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:05:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:57.093 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20735/44 objects degraded (47125.000%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":757,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1111830528,"bytes_avail":288798461952,"bytes_total":289910292480,"degraded_objects":20735,"degraded_total":44,"degraded_ratio":471.25,"recovering_objects_per_sec":110,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":882,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:05:57.113 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:05:57.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:05:57.416 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:05:57.418 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:05:57.430 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-18T12:05:57.292143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21859,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089932,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025588,"statfs":{"total":289910292480,"available":288794202112,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.635501"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":20861,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:54.056089+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:05:54.056089+0000","last_peered":"2024-09-18T12:05:54.056089+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:54.056089+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2984}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21858,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051728,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.42099999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.46100000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855491,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302464,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069376,"statfs":{"total":96636764160,"available":96327041024,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.496}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.46100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.436},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.502},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.48999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:00.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:00.752 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:00.754 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:00.765 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-18T12:05:59.292721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22223,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089932,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025588,"statfs":{"total":289910292480,"available":288794202112,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.632711"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":21589,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:59.058800+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:05:59.058800+0000","last_peered":"2024-09-18T12:05:59.058800+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:59.058800+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2620}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22222,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051729,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.52300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855491,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302464,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069376,"statfs":{"total":96636764160,"available":96327041024,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.496}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.46100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.436},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.502},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.48999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:00.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:01.065 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:01.067 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:01.077 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-18T12:05:59.292721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22223,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089932,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025588,"statfs":{"total":289910292480,"available":288794202112,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.632711"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":21589,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:59.058800+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:05:59.058800+0000","last_peered":"2024-09-18T12:05:59.058800+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:59.058800+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2620}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22222,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051729,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.52300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855491,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302464,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069376,"statfs":{"total":96636764160,"available":96327041024,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.496}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.46100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.436},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.502},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.48999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:01.079 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:01.079 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19617, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.664580+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:47.664580+0000', 'last_peered': '2024-09-18T12:05:47.664580+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:47.664580+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:01.079 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:06:01.079 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 21589, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:59.058800+0000', 'last_change': '2024-09-18T12:05:46.861100+0000', 'last_active': '2024-09-18T12:05:59.058800+0000', 'last_peered': '2024-09-18T12:05:59.058800+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:05:59.058800+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2620, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 917, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2620}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:01.079 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:01.079 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19514, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662385+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:47.662385+0000', 'last_peered': '2024-09-18T12:05:47.662385+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:47.662385+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:01.080 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:01.080 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19469, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663597+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:47.663597+0000', 'last_peered': '2024-09-18T12:05:47.663597+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:47.663597+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:01.080 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:01.080 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26917, 'reported_epoch': 48, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:47.657898+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:47.657898+0000', 'last_peered': '2024-09-18T12:05:47.657898+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:47.657898+0000', 'last_undegraded': '2024-09-18T12:05:47.657898+0000', 'last_fullsized': '2024-09-18T12:05:47.657898+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:01.080 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:01.080 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19620, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663179+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:47.663179+0000', 'last_peered': '2024-09-18T12:05:47.663179+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:47.663179+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:01.080 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:01.080 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19607, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662016+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:47.662016+0000', 'last_peered': '2024-09-18T12:05:47.662016+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:47.662016+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:01.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:01.441 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:01.441 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20371/44 objects degraded (46297.727%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":761,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1116090368,"bytes_avail":288794202112,"bytes_total":289910292480,"degraded_objects":20371,"degraded_total":44,"degraded_ratio":462.97727272727275,"recovering_objects_per_sec":117,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1246,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:01.453 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:01.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:01.731 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:01.733 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:01.743 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-18T12:06:01.293353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22223,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1089932,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":918450,"kb_avail":282025588,"statfs":{"total":289910292480,"available":288794202112,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":940493330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003738"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":21589,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:59.058800+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:05:59.058800+0000","last_peered":"2024-09-18T12:05:59.058800+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:59.058800+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2620}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22222,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051729,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.52300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855491,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302464,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94069376,"statfs":{"total":96636764160,"available":96327041024,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":309259782},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.496}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.46100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247956,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:04.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:05.035 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:05.045 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-09-18T12:06:03.293824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22223,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094092,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021428,"statfs":{"total":289910292480,"available":288789942272,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003463"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":21589,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:59.058800+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:05:59.058800+0000","last_peered":"2024-09-18T12:05:59.058800+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:05:59.058800+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2620}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22222,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051729,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.52300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855492,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306624,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065216,"statfs":{"total":96636764160,"available":96322781184,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.46999999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.35899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247956,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:05.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:05.343 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:05.345 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:05.355 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-18T12:06:05.294520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22617,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094092,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021428,"statfs":{"total":289910292480,"available":288789942272,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003655"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":22377,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:04.060132+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:04.060132+0000","last_peered":"2024-09-18T12:06:04.060132+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:04.060132+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22616,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051730,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.503},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.35599999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855492,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306624,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065216,"statfs":{"total":96636764160,"available":96322781184,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.46999999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.35899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:05.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:05.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19617, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.664580+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:47.664580+0000', 'last_peered': '2024-09-18T12:05:47.664580+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:47.664580+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:05.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:06:05.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 22377, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:04.060132+0000', 'last_change': '2024-09-18T12:05:46.861100+0000', 'last_active': '2024-09-18T12:06:04.060132+0000', 'last_peered': '2024-09-18T12:06:04.060132+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:06:04.060132+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2226, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1311, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:05.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:05.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19514, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662385+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:47.662385+0000', 'last_peered': '2024-09-18T12:05:47.662385+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:47.662385+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:05.357 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:05.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19469, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663597+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:47.663597+0000', 'last_peered': '2024-09-18T12:05:47.663597+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:47.663597+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:05.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:05.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26917, 'reported_epoch': 48, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:47.657898+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:47.657898+0000', 'last_peered': '2024-09-18T12:05:47.657898+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:47.657898+0000', 'last_undegraded': '2024-09-18T12:05:47.657898+0000', 'last_fullsized': '2024-09-18T12:05:47.657898+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:05.358 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:05.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19620, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663179+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:47.663179+0000', 'last_peered': '2024-09-18T12:05:47.663179+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:47.663179+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:05.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:05.359 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19607, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662016+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:47.662016+0000', 'last_peered': '2024-09-18T12:05:47.662016+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:47.662016+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:05.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:05.737 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:05.737 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19977/44 objects degraded (45402.273%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":765,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1120350208,"bytes_avail":288789942272,"bytes_total":289910292480,"degraded_objects":19977,"degraded_total":44,"degraded_ratio":454.02272727272725,"recovering_objects_per_sec":92,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1109,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:05.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:05.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:06.068 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:06.071 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:06.085 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-18T12:06:05.294520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22617,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094092,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021428,"statfs":{"total":289910292480,"available":288789942272,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003655"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":22377,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:04.060132+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:04.060132+0000","last_peered":"2024-09-18T12:06:04.060132+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:04.060132+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22616,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051730,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.503},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.35599999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855492,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306624,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065216,"statfs":{"total":96636764160,"available":96322781184,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.46999999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.35899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:09.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:09.385 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:09.387 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:09.397 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-09-18T12:06:09.295728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23029,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094092,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021428,"statfs":{"total":289910292480,"available":288789942272,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003584"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":23201,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:09.057756+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:09.057756+0000","last_peered":"2024-09-18T12:06:09.057756+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:09.057756+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1723,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1814}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23028,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051731,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.46700000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.38100000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855493,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306624,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065216,"statfs":{"total":96636764160,"available":96322781184,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.42099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.52900000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:09.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:09.711 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:09.713 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:09.728 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-09-18T12:06:09.295728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23029,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094092,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021428,"statfs":{"total":289910292480,"available":288789942272,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003584"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":23201,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:09.057756+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:09.057756+0000","last_peered":"2024-09-18T12:06:09.057756+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:09.057756+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1723,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1814}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23028,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051731,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.46700000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.38100000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855493,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306624,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065216,"statfs":{"total":96636764160,"available":96322781184,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.42099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.52900000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:09.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:09.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19617, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.664580+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:47.664580+0000', 'last_peered': '2024-09-18T12:05:47.664580+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:47.664580+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:09.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:06:09.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 23201, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:09.057756+0000', 'last_change': '2024-09-18T12:05:46.861100+0000', 'last_active': '2024-09-18T12:06:09.057756+0000', 'last_peered': '2024-09-18T12:06:09.057756+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:06:09.057756+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1814, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1723, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1814}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:09.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:09.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19514, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662385+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:47.662385+0000', 'last_peered': '2024-09-18T12:05:47.662385+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:47.662385+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:09.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:09.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19469, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663597+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:47.663597+0000', 'last_peered': '2024-09-18T12:05:47.663597+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:47.663597+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:09.730 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:09.731 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26917, 'reported_epoch': 48, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:47.657898+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:47.657898+0000', 'last_peered': '2024-09-18T12:05:47.657898+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:47.657898+0000', 'last_undegraded': '2024-09-18T12:05:47.657898+0000', 'last_fullsized': '2024-09-18T12:05:47.657898+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:09.731 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:09.731 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19620, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663179+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:47.663179+0000', 'last_peered': '2024-09-18T12:05:47.663179+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:47.663179+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:09.731 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:09.731 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19607, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662016+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:47.662016+0000', 'last_peered': '2024-09-18T12:05:47.662016+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:47.662016+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:09.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:10.105 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:10.106 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19565/44 objects degraded (44465.909%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":770,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1120350208,"bytes_avail":288789942272,"bytes_total":289910292480,"degraded_objects":19565,"degraded_total":44,"degraded_ratio":444.65909090909093,"recovering_objects_per_sec":97,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1170,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:10.121 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:10.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:10.416 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:10.419 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:10.430 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-09-18T12:06:09.295728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23029,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1094092,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":922610,"kb_avail":282021428,"statfs":{"total":289910292480,"available":288789942272,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":944753170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003584"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":23201,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:09.057756+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:09.057756+0000","last_peered":"2024-09-18T12:06:09.057756+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:09.057756+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1723,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1814}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23028,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051731,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.46700000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.38100000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855493,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306624,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94065216,"statfs":{"total":96636764160,"available":96322781184,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.42099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.52900000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:13.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:13.735 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:13.737 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:13.747 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-18T12:06:13.296893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23029,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1386636,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":1215154,"kb_avail":281728884,"statfs":{"total":289910292480,"available":288490377216,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1244318226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":806,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003539"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":23201,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:09.057756+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:09.057756+0000","last_peered":"2024-09-18T12:06:09.057756+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:09.057756+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1723,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1814}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23028,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051731,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.46700000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.38100000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855494,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599168,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93772672,"statfs":{"total":96636764160,"available":96023216128,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":613084678},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.47599999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.51800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.375}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.52800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:13.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:14.027 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:14.029 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:14.040 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-18T12:06:13.296893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23029,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1386636,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":1215154,"kb_avail":281728884,"statfs":{"total":289910292480,"available":288490377216,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1244318226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":806,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003539"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":23201,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:09.057756+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:09.057756+0000","last_peered":"2024-09-18T12:06:09.057756+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:09.057756+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1723,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1814}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23028,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051731,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.46700000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.38100000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855494,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599168,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93772672,"statfs":{"total":96636764160,"available":96023216128,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":613084678},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.47599999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.51800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.375}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.52800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:14.041 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:14.041 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19617, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.664580+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:47.664580+0000', 'last_peered': '2024-09-18T12:05:47.664580+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:47.664580+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:14.041 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:06:14.041 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 23201, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:09.057756+0000', 'last_change': '2024-09-18T12:05:46.861100+0000', 'last_active': '2024-09-18T12:06:09.057756+0000', 'last_peered': '2024-09-18T12:06:09.057756+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:06:09.057756+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1814, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1723, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1814}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:14.041 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:14.042 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19514, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662385+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:47.662385+0000', 'last_peered': '2024-09-18T12:05:47.662385+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:47.662385+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:14.042 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:14.042 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19469, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663597+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:47.663597+0000', 'last_peered': '2024-09-18T12:05:47.663597+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:47.663597+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:14.042 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:14.042 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26917, 'reported_epoch': 48, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:47.657898+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:47.657898+0000', 'last_peered': '2024-09-18T12:05:47.657898+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:47.657898+0000', 'last_undegraded': '2024-09-18T12:05:47.657898+0000', 'last_fullsized': '2024-09-18T12:05:47.657898+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:14.042 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:14.042 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19620, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663179+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:47.663179+0000', 'last_peered': '2024-09-18T12:05:47.663179+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:47.663179+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:14.042 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:14.042 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19607, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662016+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:47.662016+0000', 'last_peered': '2024-09-18T12:05:47.662016+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:47.662016+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:14.042 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:14.405 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:14.405 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19565/44 objects degraded (44465.909%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":774,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1419915264,"bytes_avail":288490377216,"bytes_total":289910292480,"degraded_objects":19565,"degraded_total":44,"degraded_ratio":444.65909090909093,"recovering_objects_per_sec":67,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":806,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:14.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:14.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:14.724 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:14.726 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:14.736 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-18T12:06:13.296893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23029,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1386636,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":1215154,"kb_avail":281728884,"statfs":{"total":289910292480,"available":288490377216,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1244318226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":806,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003539"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":23201,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:09.057756+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:09.057756+0000","last_peered":"2024-09-18T12:06:09.057756+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:09.057756+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1723,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1814}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23028,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051731,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.46700000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.38100000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855494,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599168,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93772672,"statfs":{"total":96636764160,"available":96023216128,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":613084678},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.47599999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.51800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.375}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.52800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:17.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:18.041 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:18.043 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:18.054 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-18T12:06:17.298048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23492,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1374156,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":1202674,"kb_avail":281741364,"statfs":{"total":289910292480,"available":288503156736,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1231538706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":875,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003527"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":24127,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:14.058463+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:14.058463+0000","last_peered":"2024-09-18T12:06:14.058463+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:14.058463+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1351}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23491,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051732,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.40799999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.501},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.48399999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855495,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586688,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93785152,"statfs":{"total":96636764160,"available":96035995648,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":600305158},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.496},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.59099999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.44},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:18.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:18.342 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:18.352 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-18T12:06:17.298048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23492,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1374156,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":1202674,"kb_avail":281741364,"statfs":{"total":289910292480,"available":288503156736,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1231538706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":875,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003527"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":24127,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:14.058463+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:14.058463+0000","last_peered":"2024-09-18T12:06:14.058463+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:14.058463+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1351}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23491,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051732,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.40799999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.501},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.48399999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855495,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586688,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93785152,"statfs":{"total":96636764160,"available":96035995648,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":600305158},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.496},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.59099999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.44},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:18.353 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:18.353 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19617, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.664580+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:47.664580+0000', 'last_peered': '2024-09-18T12:05:47.664580+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:47.664580+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:18.353 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:06:18.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 24127, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:14.058463+0000', 'last_change': '2024-09-18T12:05:46.861100+0000', 'last_active': '2024-09-18T12:06:14.058463+0000', 'last_peered': '2024-09-18T12:06:14.058463+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:06:14.058463+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1351, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2186, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1351}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:18.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:18.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19514, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662385+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:47.662385+0000', 'last_peered': '2024-09-18T12:05:47.662385+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:47.662385+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:18.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:18.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19469, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663597+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:47.663597+0000', 'last_peered': '2024-09-18T12:05:47.663597+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:47.663597+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:18.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:18.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26917, 'reported_epoch': 48, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:47.657898+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:47.657898+0000', 'last_peered': '2024-09-18T12:05:47.657898+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:47.657898+0000', 'last_undegraded': '2024-09-18T12:05:47.657898+0000', 'last_fullsized': '2024-09-18T12:05:47.657898+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:18.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:18.354 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19620, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663179+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:47.663179+0000', 'last_peered': '2024-09-18T12:05:47.663179+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:47.663179+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:18.355 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:18.355 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19607, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662016+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:47.662016+0000', 'last_peered': '2024-09-18T12:05:47.662016+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:47.662016+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:18.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:18.742 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:18.742 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19102/44 objects degraded (43413.636%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":778,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1407135744,"bytes_avail":288503156736,"bytes_total":289910292480,"degraded_objects":19102,"degraded_total":44,"degraded_ratio":434.13636363636363,"recovering_objects_per_sec":72,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":875,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:18.759 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:18.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:19.042 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:19.043 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:19.053 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-18T12:06:17.298048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23492,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1374156,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":1202674,"kb_avail":281741364,"statfs":{"total":289910292480,"available":288503156736,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1231538706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":875,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003527"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":24127,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:14.058463+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:14.058463+0000","last_peered":"2024-09-18T12:06:14.058463+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:14.058463+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1351}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23491,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051732,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.40799999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.501},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.48399999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855495,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586688,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93785152,"statfs":{"total":96636764160,"available":96035995648,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":600305158},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.496},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.59099999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.44},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:22.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:22.352 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:22.365 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-09-18T12:06:21.299335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24021,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1374156,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":1202674,"kb_avail":281741364,"statfs":{"total":289910292480,"available":288503156736,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1231538706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":992,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003607"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":25185,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:19.066308+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:19.066308+0000","last_peered":"2024-09-18T12:06:19.066308+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:19.066308+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2715,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-822}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24020,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051733,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.51500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855495,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586688,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93785152,"statfs":{"total":96636764160,"available":96035995648,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":600305158},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.496},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.59099999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:22.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:22.683 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:22.693 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-09-18T12:06:21.299335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24021,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1374156,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":1202674,"kb_avail":281741364,"statfs":{"total":289910292480,"available":288503156736,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1231538706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":992,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003607"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":25185,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:19.066308+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:19.066308+0000","last_peered":"2024-09-18T12:06:19.066308+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:19.066308+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2715,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-822}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24020,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051733,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.51500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855495,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586688,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93785152,"statfs":{"total":96636764160,"available":96035995648,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":600305158},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.496},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.59099999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:22.694 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:22.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19617, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.664580+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:47.664580+0000', 'last_peered': '2024-09-18T12:05:47.664580+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:47.664580+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:22.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:06:22.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 25185, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:19.066308+0000', 'last_change': '2024-09-18T12:05:46.861100+0000', 'last_active': '2024-09-18T12:06:19.066308+0000', 'last_peered': '2024-09-18T12:06:19.066308+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:06:19.066308+0000', 'last_undegraded': '2024-09-18T11:55:04.289074+0000', 'last_fullsized': '2024-09-18T11:55:04.247003+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 822, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2715, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -822}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:22.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:22.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19514, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662385+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:47.662385+0000', 'last_peered': '2024-09-18T12:05:47.662385+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:47.662385+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:22.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:22.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19469, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663597+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:47.663597+0000', 'last_peered': '2024-09-18T12:05:47.663597+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:47.663597+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:22.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:22.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26917, 'reported_epoch': 48, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:47.657898+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:47.657898+0000', 'last_peered': '2024-09-18T12:05:47.657898+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:47.657898+0000', 'last_undegraded': '2024-09-18T12:05:47.657898+0000', 'last_fullsized': '2024-09-18T12:05:47.657898+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:22.695 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:22.696 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19620, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663179+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:47.663179+0000', 'last_peered': '2024-09-18T12:05:47.663179+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:47.663179+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:22.696 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:22.696 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19607, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662016+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:47.662016+0000', 'last_peered': '2024-09-18T12:05:47.662016+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:47.662016+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:22.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:23.072 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:23.072 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18573/44 objects degraded (42211.364%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":783,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1407135744,"bytes_avail":288503156736,"bytes_total":289910292480,"degraded_objects":18573,"degraded_total":44,"degraded_ratio":422.11363636363637,"recovering_objects_per_sec":82,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":992,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:23.091 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:23.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:23.394 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:23.396 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:23.408 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-09-18T12:06:23.300117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24021,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1378316,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":1206834,"kb_avail":281737204,"statfs":{"total":289910292480,"available":288498896896,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1235798546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":992,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003900"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":25185,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:19.066308+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:19.066308+0000","last_peered":"2024-09-18T12:06:19.066308+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:19.066308+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2715,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-822}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24020,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051733,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:05:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.5549999999999999,"15min":1.774},"min":{"1min":0.28899999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":5.1820000000000004,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3420000000000001,"15min":1.1180000000000001},"min":{"1min":0.23799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.415,"5min":12.712,"15min":12.712},"last":0.51500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855496,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590848,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93780992,"statfs":{"total":96636764160,"available":96031735808,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":604564998},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.496},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.59099999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:26.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:26.717 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:26.728 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2024-09-18T12:06:25.864719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24630,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1378316,"kb_used_data":171428,"kb_used_omap":13,"kb_used_meta":1206834,"kb_avail":281737204,"statfs":{"total":289910292480,"available":288498896896,"internally_reserved":0,"allocated":175542272,"data_stored":174677936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1235798546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.567208"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":26403,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:24.067464+0000","last_change":"2024-09-18T12:05:46.861100+0000","last_active":"2024-09-18T12:06:24.067464+0000","last_peered":"2024-09-18T12:06:24.067464+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:24.067464+0000","last_undegraded":"2024-09-18T11:55:04.289074+0000","last_fullsized":"2024-09-18T11:55:04.247003+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:12:38.747654+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3324,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-213}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663332+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:47.663332+0000","last_peered":"2024-09-18T12:05:47.663332+0000","last_clean":"2024-09-18T12:05:47.663332+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:47.663332+0000","last_undegraded":"2024-09-18T12:05:47.663332+0000","last_fullsized":"2024-09-18T12:05:47.663332+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26543,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663807+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:47.663807+0000","last_peered":"2024-09-18T12:05:47.663807+0000","last_clean":"2024-09-18T12:05:47.663807+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:47.663807+0000","last_undegraded":"2024-09-18T12:05:47.663807+0000","last_fullsized":"2024-09-18T12:05:47.663807+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663729+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:47.663729+0000","last_peered":"2024-09-18T12:05:47.663729+0000","last_clean":"2024-09-18T12:05:47.663729+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:47.663729+0000","last_undegraded":"2024-09-18T12:05:47.663729+0000","last_fullsized":"2024-09-18T12:05:47.663729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663962+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:47.663962+0000","last_peered":"2024-09-18T12:05:47.663962+0000","last_clean":"2024-09-18T12:05:47.663962+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:47.663962+0000","last_undegraded":"2024-09-18T12:05:47.663962+0000","last_fullsized":"2024-09-18T12:05:47.663962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26678,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664057+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:47.664057+0000","last_peered":"2024-09-18T12:05:47.664057+0000","last_clean":"2024-09-18T12:05:47.664057+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:47.664057+0000","last_undegraded":"2024-09-18T12:05:47.664057+0000","last_fullsized":"2024-09-18T12:05:47.664057+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19296,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.744541+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.744541+0000","last_peered":"2024-09-18T12:05:48.744541+0000","last_clean":"2024-09-18T12:05:48.744541+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.744541+0000","last_undegraded":"2024-09-18T12:05:48.744541+0000","last_fullsized":"2024-09-18T12:05:48.744541+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.663952+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:47.663952+0000","last_peered":"2024-09-18T12:05:47.663952+0000","last_clean":"2024-09-18T12:05:47.663952+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:47.663952+0000","last_undegraded":"2024-09-18T12:05:47.663952+0000","last_fullsized":"2024-09-18T12:05:47.663952+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.664047+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:47.664047+0000","last_peered":"2024-09-18T12:05:47.664047+0000","last_clean":"2024-09-18T12:05:47.664047+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:47.664047+0000","last_undegraded":"2024-09-18T12:05:47.664047+0000","last_fullsized":"2024-09-18T12:05:47.664047+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24629,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174305280,"data_stored":174267011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051734,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38500000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.44},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.32300000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855496,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590848,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93780992,"statfs":{"total":96636764160,"available":96031735808,"internally_reserved":0,"allocated":450560,"data_stored":142686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":604564998},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.496},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.59099999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.44600000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:26.730 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:27.113 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:27.115 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:27.126 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-18T12:06:26.866145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21306,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1382476,"kb_used_data":171424,"kb_used_omap":13,"kb_used_meta":1210994,"kb_avail":281733044,"statfs":{"total":289910292480,"available":288494637056,"internally_reserved":0,"allocated":175538176,"data_stored":174678928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1240058386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.568097"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19753,"reported_epoch":50,"state":"peering","last_fresh":"2024-09-18T12:06:25.879624+0000","last_change":"2024-09-18T12:06:25.874133+0000","last_active":"2024-09-18T12:06:25.873931+0000","last_peered":"2024-09-18T12:04:30.900026+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:25.879624+0000","last_undegraded":"2024-09-18T12:06:25.879624+0000","last_fullsized":"2024-09-18T12:06:25.879624+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:18:58.670834+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.666852+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:48.666852+0000","last_peered":"2024-09-18T12:05:48.666852+0000","last_clean":"2024-09-18T12:05:48.666852+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:48.666852+0000","last_undegraded":"2024-09-18T12:05:48.666852+0000","last_fullsized":"2024-09-18T12:05:48.666852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26544,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667607+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:48.667607+0000","last_peered":"2024-09-18T12:05:48.667607+0000","last_clean":"2024-09-18T12:05:48.667607+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:48.667607+0000","last_undegraded":"2024-09-18T12:05:48.667607+0000","last_fullsized":"2024-09-18T12:05:48.667607+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":82,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667655+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:48.667655+0000","last_peered":"2024-09-18T12:05:48.667655+0000","last_clean":"2024-09-18T12:05:48.667655+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:48.667655+0000","last_undegraded":"2024-09-18T12:05:48.667655+0000","last_fullsized":"2024-09-18T12:05:48.667655+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667165+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:48.667165+0000","last_peered":"2024-09-18T12:05:48.667165+0000","last_clean":"2024-09-18T12:05:48.667165+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:48.667165+0000","last_undegraded":"2024-09-18T12:05:48.667165+0000","last_fullsized":"2024-09-18T12:05:48.667165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26679,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667612+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:48.667612+0000","last_peered":"2024-09-18T12:05:48.667612+0000","last_clean":"2024-09-18T12:05:48.667612+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:48.667612+0000","last_undegraded":"2024-09-18T12:05:48.667612+0000","last_fullsized":"2024-09-18T12:05:48.667612+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19297,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.760276+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.760276+0000","last_peered":"2024-09-18T12:05:48.760276+0000","last_clean":"2024-09-18T12:05:48.760276+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.760276+0000","last_undegraded":"2024-09-18T12:05:48.760276+0000","last_fullsized":"2024-09-18T12:05:48.760276+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667567+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:48.667567+0000","last_peered":"2024-09-18T12:05:48.667567+0000","last_clean":"2024-09-18T12:05:48.667567+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:48.667567+0000","last_undegraded":"2024-09-18T12:05:48.667567+0000","last_fullsized":"2024-09-18T12:05:48.667567+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667692+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:48.667692+0000","last_peered":"2024-09-18T12:05:48.667692+0000","last_clean":"2024-09-18T12:05:48.667692+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:48.667692+0000","last_undegraded":"2024-09-18T12:05:48.667692+0000","last_fullsized":"2024-09-18T12:05:48.667692+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21305,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051734,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38500000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.44},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.32300000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855497,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595008,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93776832,"statfs":{"total":96636764160,"available":96027475968,"internally_reserved":0,"allocated":446464,"data_stored":143678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":608824838},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.503}]}]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.44600000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:27.128 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:27.128 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19617, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.664580+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:05:47.664580+0000', 'last_peered': '2024-09-18T12:05:47.664580+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:05:47.664580+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:27.128 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:06:27.128 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19753, 'reported_epoch': 50, 'state': 'peering', 'last_fresh': '2024-09-18T12:06:25.879624+0000', 'last_change': '2024-09-18T12:06:25.874133+0000', 'last_active': '2024-09-18T12:06:25.873931+0000', 'last_peered': '2024-09-18T12:04:30.900026+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:06:25.879624+0000', 'last_undegraded': '2024-09-18T12:06:25.879624+0000', 'last_fullsized': '2024-09-18T12:06:25.879624+0000', 'mapping_epoch': 50, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:18:58.670834+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-18T12:06:27.128 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:27.128 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19514, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662385+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:05:47.662385+0000', 'last_peered': '2024-09-18T12:05:47.662385+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:05:47.662385+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:27.128 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:27.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19469, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663597+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:05:47.663597+0000', 'last_peered': '2024-09-18T12:05:47.663597+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:05:47.663597+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:27.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:27.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26917, 'reported_epoch': 48, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:05:47.657898+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:05:47.657898+0000', 'last_peered': '2024-09-18T12:05:47.657898+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:05:47.657898+0000', 'last_undegraded': '2024-09-18T12:05:47.657898+0000', 'last_fullsized': '2024-09-18T12:05:47.657898+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:27.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:27.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19620, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.663179+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:05:47.663179+0000', 'last_peered': '2024-09-18T12:05:47.663179+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:05:47.663179+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:27.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:27.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 19607, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:05:47.662016+0000', 'last_change': '2024-09-18T11:55:04.739063+0000', 'last_active': '2024-09-18T12:05:47.662016+0000', 'last_peered': '2024-09-18T12:05:47.662016+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:05:47.662016+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3526, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:27.129 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:27.573 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:27.574 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17964/44 objects degraded (40827.273%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":787,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":51,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1411395584,"bytes_avail":288498896896,"bytes_total":289910292480,"degraded_objects":17964,"degraded_total":44,"degraded_ratio":408.27272727272725,"recovering_objects_per_sec":133,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1601,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:27.588 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:27.588 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:27.892 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:27.894 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:27.906 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-18T12:06:27.301310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21306,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1382476,"kb_used_data":171424,"kb_used_omap":13,"kb_used_meta":1210994,"kb_avail":281733044,"statfs":{"total":289910292480,"available":288494637056,"internally_reserved":0,"allocated":175538176,"data_stored":174678928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1240058386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2715,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002721"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19600,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658854+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:05:47.658854+0000","last_peered":"2024-09-18T12:05:47.658854+0000","last_clean":"2024-09-18T12:05:47.658854+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:05:47.658854+0000","last_undegraded":"2024-09-18T12:05:47.658854+0000","last_fullsized":"2024-09-18T12:05:47.658854+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19753,"reported_epoch":50,"state":"peering","last_fresh":"2024-09-18T12:06:25.879624+0000","last_change":"2024-09-18T12:06:25.874133+0000","last_active":"2024-09-18T12:06:25.873931+0000","last_peered":"2024-09-18T12:04:30.900026+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:25.879624+0000","last_undegraded":"2024-09-18T12:06:25.879624+0000","last_fullsized":"2024-09-18T12:06:25.879624+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:18:58.670834+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19299,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.659666+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:05:47.659666+0000","last_peered":"2024-09-18T12:05:47.659666+0000","last_clean":"2024-09-18T12:05:47.659666+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:05:47.659666+0000","last_undegraded":"2024-09-18T12:05:47.659666+0000","last_fullsized":"2024-09-18T12:05:47.659666+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.666852+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:48.666852+0000","last_peered":"2024-09-18T12:05:48.666852+0000","last_clean":"2024-09-18T12:05:48.666852+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:48.666852+0000","last_undegraded":"2024-09-18T12:05:48.666852+0000","last_fullsized":"2024-09-18T12:05:48.666852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26544,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667607+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:48.667607+0000","last_peered":"2024-09-18T12:05:48.667607+0000","last_clean":"2024-09-18T12:05:48.667607+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:48.667607+0000","last_undegraded":"2024-09-18T12:05:48.667607+0000","last_fullsized":"2024-09-18T12:05:48.667607+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658010+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:05:47.658010+0000","last_peered":"2024-09-18T12:05:47.658010+0000","last_clean":"2024-09-18T12:05:47.658010+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:05:47.658010+0000","last_undegraded":"2024-09-18T12:05:47.658010+0000","last_fullsized":"2024-09-18T12:05:47.658010+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26644,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.658108+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:05:47.658108+0000","last_peered":"2024-09-18T12:05:47.658108+0000","last_clean":"2024-09-18T12:05:47.658108+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:05:47.658108+0000","last_undegraded":"2024-09-18T12:05:47.658108+0000","last_fullsized":"2024-09-18T12:05:47.658108+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":82,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667655+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:48.667655+0000","last_peered":"2024-09-18T12:05:48.667655+0000","last_clean":"2024-09-18T12:05:48.667655+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:48.667655+0000","last_undegraded":"2024-09-18T12:05:48.667655+0000","last_fullsized":"2024-09-18T12:05:48.667655+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26917,"reported_epoch":48,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:05:47.657898+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:05:47.657898+0000","last_peered":"2024-09-18T12:05:47.657898+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:05:47.657898+0000","last_undegraded":"2024-09-18T12:05:47.657898+0000","last_fullsized":"2024-09-18T12:05:47.657898+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667165+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:48.667165+0000","last_peered":"2024-09-18T12:05:48.667165+0000","last_clean":"2024-09-18T12:05:48.667165+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:48.667165+0000","last_undegraded":"2024-09-18T12:05:48.667165+0000","last_fullsized":"2024-09-18T12:05:48.667165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26679,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667612+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:48.667612+0000","last_peered":"2024-09-18T12:05:48.667612+0000","last_clean":"2024-09-18T12:05:48.667612+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:48.667612+0000","last_undegraded":"2024-09-18T12:05:48.667612+0000","last_fullsized":"2024-09-18T12:05:48.667612+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":49,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.668026+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:05:47.668026+0000","last_peered":"2024-09-18T12:05:47.668026+0000","last_clean":"2024-09-18T12:05:47.668026+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:05:47.668026+0000","last_undegraded":"2024-09-18T12:05:47.668026+0000","last_fullsized":"2024-09-18T12:05:47.668026+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19297,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.760276+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.760276+0000","last_peered":"2024-09-18T12:05:48.760276+0000","last_clean":"2024-09-18T12:05:48.760276+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.760276+0000","last_undegraded":"2024-09-18T12:05:48.760276+0000","last_fullsized":"2024-09-18T12:05:48.760276+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667567+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:48.667567+0000","last_peered":"2024-09-18T12:05:48.667567+0000","last_clean":"2024-09-18T12:05:48.667567+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:48.667567+0000","last_undegraded":"2024-09-18T12:05:48.667567+0000","last_fullsized":"2024-09-18T12:05:48.667567+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667692+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:48.667692+0000","last_peered":"2024-09-18T12:05:48.667692+0000","last_clean":"2024-09-18T12:05:48.667692+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:48.667692+0000","last_undegraded":"2024-09-18T12:05:48.667692+0000","last_fullsized":"2024-09-18T12:05:48.667692+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21305,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051734,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420152,"kb_used_data":113960,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951688,"statfs":{"total":96636764160,"available":96206528512,"internally_reserved":0,"allocated":116695040,"data_stored":116418620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38500000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.44},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.32300000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855497,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595008,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93776832,"statfs":{"total":96636764160,"available":96027475968,"internally_reserved":0,"allocated":446464,"data_stored":143678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":608824838},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.503}]}]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.44600000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:30.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:31.250 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:31.252 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:31.268 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2024-09-18T12:06:29.301733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21306,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1382492,"kb_used_data":171440,"kb_used_omap":13,"kb_used_meta":1210994,"kb_avail":281733028,"statfs":{"total":289910292480,"available":288494620672,"internally_reserved":0,"allocated":175554560,"data_stored":174684593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1240058386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2715,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002398"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19617,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.664580+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:05:47.664580+0000","last_peered":"2024-09-18T12:05:47.664580+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:05:47.664580+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19753,"reported_epoch":50,"state":"peering","last_fresh":"2024-09-18T12:06:25.879624+0000","last_change":"2024-09-18T12:06:25.874133+0000","last_active":"2024-09-18T12:06:25.873931+0000","last_peered":"2024-09-18T12:04:30.900026+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:25.879624+0000","last_undegraded":"2024-09-18T12:06:25.879624+0000","last_fullsized":"2024-09-18T12:06:25.879624+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:18:58.670834+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19648,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661962+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:05:47.661962+0000","last_peered":"2024-09-18T12:05:47.661962+0000","last_clean":"2024-09-18T12:05:47.661962+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:05:47.661962+0000","last_undegraded":"2024-09-18T12:05:47.661962+0000","last_fullsized":"2024-09-18T12:05:47.661962+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662385+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:05:47.662385+0000","last_peered":"2024-09-18T12:05:47.662385+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:05:47.662385+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663597+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:05:47.663597+0000","last_peered":"2024-09-18T12:05:47.663597+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:05:47.663597+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19837,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661722+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:05:47.661722+0000","last_peered":"2024-09-18T12:05:47.661722+0000","last_clean":"2024-09-18T12:05:47.661722+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:05:47.661722+0000","last_undegraded":"2024-09-18T12:05:47.661722+0000","last_fullsized":"2024-09-18T12:05:47.661722+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.666852+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:48.666852+0000","last_peered":"2024-09-18T12:05:48.666852+0000","last_clean":"2024-09-18T12:05:48.666852+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:48.666852+0000","last_undegraded":"2024-09-18T12:05:48.666852+0000","last_fullsized":"2024-09-18T12:05:48.666852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26544,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667607+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:48.667607+0000","last_peered":"2024-09-18T12:05:48.667607+0000","last_clean":"2024-09-18T12:05:48.667607+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:48.667607+0000","last_undegraded":"2024-09-18T12:05:48.667607+0000","last_fullsized":"2024-09-18T12:05:48.667607+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26613,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661919+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:05:47.661919+0000","last_peered":"2024-09-18T12:05:47.661919+0000","last_clean":"2024-09-18T12:05:47.661919+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:05:47.661919+0000","last_undegraded":"2024-09-18T12:05:47.661919+0000","last_fullsized":"2024-09-18T12:05:47.661919+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-18T12:05:47.661703+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:05:47.661703+0000","last_peered":"2024-09-18T12:05:47.661703+0000","last_clean":"2024-09-18T12:05:47.661703+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:05:47.661703+0000","last_undegraded":"2024-09-18T12:05:47.661703+0000","last_fullsized":"2024-09-18T12:05:47.661703+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":82,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667655+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:48.667655+0000","last_peered":"2024-09-18T12:05:48.667655+0000","last_clean":"2024-09-18T12:05:48.667655+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:48.667655+0000","last_undegraded":"2024-09-18T12:05:48.667655+0000","last_fullsized":"2024-09-18T12:05:48.667655+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.663179+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:05:47.663179+0000","last_peered":"2024-09-18T12:05:47.663179+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:05:47.663179+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667165+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:48.667165+0000","last_peered":"2024-09-18T12:05:48.667165+0000","last_clean":"2024-09-18T12:05:48.667165+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:48.667165+0000","last_undegraded":"2024-09-18T12:05:48.667165+0000","last_fullsized":"2024-09-18T12:05:48.667165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26679,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667612+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:48.667612+0000","last_peered":"2024-09-18T12:05:48.667612+0000","last_clean":"2024-09-18T12:05:48.667612+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:48.667612+0000","last_undegraded":"2024-09-18T12:05:48.667612+0000","last_fullsized":"2024-09-18T12:05:48.667612+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19297,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.760276+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.760276+0000","last_peered":"2024-09-18T12:05:48.760276+0000","last_clean":"2024-09-18T12:05:48.760276+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.760276+0000","last_undegraded":"2024-09-18T12:05:48.760276+0000","last_fullsized":"2024-09-18T12:05:48.760276+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667567+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:48.667567+0000","last_peered":"2024-09-18T12:05:48.667567+0000","last_clean":"2024-09-18T12:05:48.667567+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:48.667567+0000","last_undegraded":"2024-09-18T12:05:48.667567+0000","last_fullsized":"2024-09-18T12:05:48.667567+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667692+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:48.667692+0000","last_peered":"2024-09-18T12:05:48.667692+0000","last_clean":"2024-09-18T12:05:48.667692+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:48.667692+0000","last_undegraded":"2024-09-18T12:05:48.667692+0000","last_fullsized":"2024-09-18T12:05:48.667692+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19607,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:05:47.662016+0000","last_change":"2024-09-18T11:55:04.739063+0000","last_active":"2024-09-18T12:05:47.662016+0000","last_peered":"2024-09-18T12:05:47.662016+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:05:47.662016+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3519}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21305,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051735,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.311},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.29399999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.26900000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855497,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595008,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93776832,"statfs":{"total":96636764160,"available":96027475968,"internally_reserved":0,"allocated":446464,"data_stored":143678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":608824838},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.503}]}]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367316,"kb_used_data":57028,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004524,"statfs":{"total":96636764160,"available":96260632576,"internally_reserved":0,"allocated":58396672,"data_stored":58116630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.44600000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:31.270 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:31.580 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:31.598 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:31.598 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-18T12:06:31.302270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21815,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1382504,"kb_used_data":171456,"kb_used_omap":13,"kb_used_meta":1210994,"kb_avail":281733016,"statfs":{"total":289910292480,"available":288494608384,"internally_reserved":0,"allocated":175570944,"data_stored":174690258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1240058386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2206,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002153"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19753,"reported_epoch":50,"state":"peering","last_fresh":"2024-09-18T12:06:25.879624+0000","last_change":"2024-09-18T12:06:25.874133+0000","last_active":"2024-09-18T12:06:25.873931+0000","last_peered":"2024-09-18T12:04:30.900026+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:25.879624+0000","last_undegraded":"2024-09-18T12:06:25.879624+0000","last_fullsized":"2024-09-18T12:06:25.879624+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:18:58.670834+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.666852+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:48.666852+0000","last_peered":"2024-09-18T12:05:48.666852+0000","last_clean":"2024-09-18T12:05:48.666852+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:48.666852+0000","last_undegraded":"2024-09-18T12:05:48.666852+0000","last_fullsized":"2024-09-18T12:05:48.666852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26544,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667607+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:48.667607+0000","last_peered":"2024-09-18T12:05:48.667607+0000","last_clean":"2024-09-18T12:05:48.667607+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:48.667607+0000","last_undegraded":"2024-09-18T12:05:48.667607+0000","last_fullsized":"2024-09-18T12:05:48.667607+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":82,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667655+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:48.667655+0000","last_peered":"2024-09-18T12:05:48.667655+0000","last_clean":"2024-09-18T12:05:48.667655+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:48.667655+0000","last_undegraded":"2024-09-18T12:05:48.667655+0000","last_fullsized":"2024-09-18T12:05:48.667655+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667165+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:48.667165+0000","last_peered":"2024-09-18T12:05:48.667165+0000","last_clean":"2024-09-18T12:05:48.667165+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:48.667165+0000","last_undegraded":"2024-09-18T12:05:48.667165+0000","last_fullsized":"2024-09-18T12:05:48.667165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26679,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667612+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:48.667612+0000","last_peered":"2024-09-18T12:05:48.667612+0000","last_clean":"2024-09-18T12:05:48.667612+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:48.667612+0000","last_undegraded":"2024-09-18T12:05:48.667612+0000","last_fullsized":"2024-09-18T12:05:48.667612+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19297,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.760276+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.760276+0000","last_peered":"2024-09-18T12:05:48.760276+0000","last_clean":"2024-09-18T12:05:48.760276+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.760276+0000","last_undegraded":"2024-09-18T12:05:48.760276+0000","last_fullsized":"2024-09-18T12:05:48.760276+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667567+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:48.667567+0000","last_peered":"2024-09-18T12:05:48.667567+0000","last_clean":"2024-09-18T12:05:48.667567+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:48.667567+0000","last_undegraded":"2024-09-18T12:05:48.667567+0000","last_fullsized":"2024-09-18T12:05:48.667567+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667692+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:48.667692+0000","last_peered":"2024-09-18T12:05:48.667692+0000","last_clean":"2024-09-18T12:05:48.667692+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:48.667692+0000","last_undegraded":"2024-09-18T12:05:48.667692+0000","last_fullsized":"2024-09-18T12:05:48.667692+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":20628,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:29.392969+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:29.392969+0000","last_peered":"2024-09-18T12:06:29.392969+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:29.392969+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":513,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21814,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051735,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.311},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.29399999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.26900000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855497,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595008,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93776832,"statfs":{"total":96636764160,"available":96027475968,"internally_reserved":0,"allocated":446464,"data_stored":143678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":608824838},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.503}]}]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.215},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.249}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.314},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.747}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:31.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:31.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19619, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.871756+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:25.871756+0000', 'last_peered': '2024-09-18T12:06:25.871756+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:25.871756+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:31.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-18T12:06:31.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "43'19558", 'reported_seq': 19753, 'reported_epoch': 50, 'state': 'peering', 'last_fresh': '2024-09-18T12:06:25.879624+0000', 'last_change': '2024-09-18T12:06:25.874133+0000', 'last_active': '2024-09-18T12:06:25.873931+0000', 'last_peered': '2024-09-18T12:04:30.900026+0000', 'last_clean': '2024-09-18T11:55:02.215905+0000', 'last_became_active': '2024-09-18T11:55:04.770553+0000', 'last_became_peered': '2024-09-18T11:55:04.770553+0000', 'last_unstale': '2024-09-18T12:06:25.879624+0000', 'last_undegraded': '2024-09-18T12:06:25.879624+0000', 'last_fullsized': '2024-09-18T12:06:25.879624+0000', 'mapping_epoch': 50, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "30'3208", 'last_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:54:55.665140+0000', 'objects_scrubbed': 3138, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:18:58.670834+0000', 'scrub_duration': 2.012298633, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16416, 'num_write_kb': 26553, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-18T12:06:31.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:31.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19516, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867475+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:25.867475+0000', 'last_peered': '2024-09-18T12:06:25.867475+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:25.867475+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:31.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:31.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19471, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867373+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:06:25.867373+0000', 'last_peered': '2024-09-18T12:06:25.867373+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:06:25.867373+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:31.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:31.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26919, 'reported_epoch': 50, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:06:25.871651+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:06:25.871651+0000', 'last_peered': '2024-09-18T12:06:25.871651+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:06:25.871651+0000', 'last_undegraded': '2024-09-18T12:06:25.871651+0000', 'last_fullsized': '2024-09-18T12:06:25.871651+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:31.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:31.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19622, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.870033+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:25.870033+0000', 'last_peered': '2024-09-18T12:06:25.870033+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:25.870033+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:31.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:31.601 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 20628, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:29.392969+0000', 'last_change': '2024-09-18T12:06:25.584319+0000', 'last_active': '2024-09-18T12:06:29.392969+0000', 'last_peered': '2024-09-18T12:06:29.392969+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:06:29.392969+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 513, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3013}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:31.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:31.983 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:31.983 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg peering","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17242/44 objects degraded (39186.364%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":791,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":51,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"peering","count":1},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1415684096,"bytes_avail":288494608384,"bytes_total":289910292480,"inactive_pgs_ratio":0.041666667908430099,"degraded_objects":17242,"degraded_total":44,"degraded_ratio":391.86363636363637},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:32.000 INFO:tasks.rep_lost_unfound_delete.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T12:06:32.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:32.300 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:32.303 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:32.316 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-18T12:06:31.302270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21815,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1382504,"kb_used_data":171456,"kb_used_omap":13,"kb_used_meta":1210994,"kb_avail":281733016,"statfs":{"total":289910292480,"available":288494608384,"internally_reserved":0,"allocated":175570944,"data_stored":174690258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1240058386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2206,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002153"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19753,"reported_epoch":50,"state":"peering","last_fresh":"2024-09-18T12:06:25.879624+0000","last_change":"2024-09-18T12:06:25.874133+0000","last_active":"2024-09-18T12:06:25.873931+0000","last_peered":"2024-09-18T12:04:30.900026+0000","last_clean":"2024-09-18T11:55:02.215905+0000","last_became_active":"2024-09-18T11:55:04.770553+0000","last_became_peered":"2024-09-18T11:55:04.770553+0000","last_unstale":"2024-09-18T12:06:25.879624+0000","last_undegraded":"2024-09-18T12:06:25.879624+0000","last_fullsized":"2024-09-18T12:06:25.879624+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:18:58.670834+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.666852+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:05:48.666852+0000","last_peered":"2024-09-18T12:05:48.666852+0000","last_clean":"2024-09-18T12:05:48.666852+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:05:48.666852+0000","last_undegraded":"2024-09-18T12:05:48.666852+0000","last_fullsized":"2024-09-18T12:05:48.666852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26544,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667607+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:05:48.667607+0000","last_peered":"2024-09-18T12:05:48.667607+0000","last_clean":"2024-09-18T12:05:48.667607+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:05:48.667607+0000","last_undegraded":"2024-09-18T12:05:48.667607+0000","last_fullsized":"2024-09-18T12:05:48.667607+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":82,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667655+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:05:48.667655+0000","last_peered":"2024-09-18T12:05:48.667655+0000","last_clean":"2024-09-18T12:05:48.667655+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:05:48.667655+0000","last_undegraded":"2024-09-18T12:05:48.667655+0000","last_fullsized":"2024-09-18T12:05:48.667655+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667165+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:05:48.667165+0000","last_peered":"2024-09-18T12:05:48.667165+0000","last_clean":"2024-09-18T12:05:48.667165+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:05:48.667165+0000","last_undegraded":"2024-09-18T12:05:48.667165+0000","last_fullsized":"2024-09-18T12:05:48.667165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26679,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667612+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:05:48.667612+0000","last_peered":"2024-09-18T12:05:48.667612+0000","last_clean":"2024-09-18T12:05:48.667612+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:05:48.667612+0000","last_undegraded":"2024-09-18T12:05:48.667612+0000","last_fullsized":"2024-09-18T12:05:48.667612+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19297,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.760276+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:05:48.760276+0000","last_peered":"2024-09-18T12:05:48.760276+0000","last_clean":"2024-09-18T12:05:48.760276+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:05:48.760276+0000","last_undegraded":"2024-09-18T12:05:48.760276+0000","last_fullsized":"2024-09-18T12:05:48.760276+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667567+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:05:48.667567+0000","last_peered":"2024-09-18T12:05:48.667567+0000","last_clean":"2024-09-18T12:05:48.667567+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:05:48.667567+0000","last_undegraded":"2024-09-18T12:05:48.667567+0000","last_fullsized":"2024-09-18T12:05:48.667567+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-18T12:05:48.667692+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:05:48.667692+0000","last_peered":"2024-09-18T12:05:48.667692+0000","last_clean":"2024-09-18T12:05:48.667692+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:05:48.667692+0000","last_undegraded":"2024-09-18T12:05:48.667692+0000","last_fullsized":"2024-09-18T12:05:48.667692+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":20628,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:29.392969+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:29.392969+0000","last_peered":"2024-09-18T12:06:29.392969+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:29.392969+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":513,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21814,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051735,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.311},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.29399999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.26900000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855497,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595008,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93776832,"statfs":{"total":96636764160,"available":96027475968,"internally_reserved":0,"allocated":446464,"data_stored":143678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":608824838},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.498,"15min":0.46000000000000002},"min":{"1min":0.29299999999999998,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.57499999999999996,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":1.2569999999999999,"15min":1.248},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.56299999999999994,"5min":9.6940000000000008,"15min":12.244999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":1.615,"15min":1.351},"min":{"1min":0.24299999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":13.737,"15min":13.737},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.83999999999999997,"15min":0.77300000000000002},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.65100000000000002,"5min":8.4730000000000008,"15min":11.996},"last":0.503}]}]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.0660000000000001,"15min":1.03},"min":{"1min":0.24299999999999999,"5min":0.17599999999999999,"15min":0.161},"max":{"1min":0.65500000000000003,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.215},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.73299999999999998,"15min":0.95499999999999996},"min":{"1min":0.26200000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.56299999999999994,"5min":5.875,"15min":8.3460000000000001},"last":0.249}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.314},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.747}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:35.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:35.635 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:35.637 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:35.648 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-18T12:06:35.303285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26066,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1702764,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281412756,"statfs":{"total":289910292480,"available":288166662144,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.438565"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":22056,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:34.394213+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:34.394213+0000","last_peered":"2024-09-18T12:06:34.394213+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:34.394213+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1227,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26065,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051736,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.32600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.36599999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855498,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915268,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456572,"statfs":{"total":96636764160,"available":95699529728,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.35099999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.41999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:35.649 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:35.955 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:35.957 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:35.967 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-18T12:06:35.303285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26066,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1702764,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281412756,"statfs":{"total":289910292480,"available":288166662144,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.438565"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":22056,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:34.394213+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:34.394213+0000","last_peered":"2024-09-18T12:06:34.394213+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:34.394213+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1227,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26065,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051736,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.32600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.36599999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855498,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915268,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456572,"statfs":{"total":96636764160,"available":95699529728,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.35099999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.41999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:35.968 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:35.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19619, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.871756+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:25.871756+0000', 'last_peered': '2024-09-18T12:06:25.871756+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:25.871756+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:35.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:35.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19516, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867475+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:25.867475+0000', 'last_peered': '2024-09-18T12:06:25.867475+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:25.867475+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:35.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:35.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19471, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867373+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:06:25.867373+0000', 'last_peered': '2024-09-18T12:06:25.867373+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:06:25.867373+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:35.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:35.969 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26919, 'reported_epoch': 50, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:06:25.871651+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:06:25.871651+0000', 'last_peered': '2024-09-18T12:06:25.871651+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:06:25.871651+0000', 'last_undegraded': '2024-09-18T12:06:25.871651+0000', 'last_fullsized': '2024-09-18T12:06:25.871651+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:35.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:35.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19622, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.870033+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:25.870033+0000', 'last_peered': '2024-09-18T12:06:25.870033+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:25.870033+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:35.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:35.970 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 22056, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:34.394213+0000', 'last_change': '2024-09-18T12:06:25.584319+0000', 'last_active': '2024-09-18T12:06:34.394213+0000', 'last_peered': '2024-09-18T12:06:34.394213+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:06:34.394213+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2303, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1227, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:35.970 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:36.324 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:36.324 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16528/44 objects degraded (37563.636%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":796,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":51,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1743630336,"bytes_avail":288166662144,"bytes_total":289910292480,"degraded_objects":16528,"degraded_total":44,"degraded_ratio":375.63636363636363,"recovering_objects_per_sec":152,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1436,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:36.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:36.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:36.651 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:36.653 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:36.666 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-18T12:06:35.303285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26066,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1702764,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281412756,"statfs":{"total":289910292480,"available":288166662144,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.438565"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":22056,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:34.394213+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:34.394213+0000","last_peered":"2024-09-18T12:06:34.394213+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:34.394213+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1227,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26065,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051736,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.98699999999999999,"15min":0.71199999999999997},"min":{"1min":0.29199999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.58099999999999996,"5min":19.776,"15min":19.776},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.437,"5min":1.609,"15min":1.143},"min":{"1min":0.28199999999999997,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.68300000000000005,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.32600000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.36599999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855498,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915268,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456572,"statfs":{"total":96636764160,"available":95699529728,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.35099999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.41999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:39.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:39.993 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:39.995 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:40.006 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-18T12:06:39.304393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26066,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1706924,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281408596,"statfs":{"total":289910292480,"available":288162402304,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4760,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003083"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":22056,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:34.394213+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:34.394213+0000","last_peered":"2024-09-18T12:06:34.394213+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:34.394213+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1227,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26065,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051737,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41899999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.505},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.46200000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855499,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919428,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452412,"statfs":{"total":96636764160,"available":95695269888,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.308},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.38}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.45900000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:40.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:40.298 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:40.301 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:40.313 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-18T12:06:39.304393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26066,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1706924,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281408596,"statfs":{"total":289910292480,"available":288162402304,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4760,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003083"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":22056,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:34.394213+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:34.394213+0000","last_peered":"2024-09-18T12:06:34.394213+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:34.394213+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1227,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26065,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051737,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41899999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.505},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.46200000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855499,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919428,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452412,"statfs":{"total":96636764160,"available":95695269888,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.308},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.38}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.45900000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:40.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:40.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19619, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.871756+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:25.871756+0000', 'last_peered': '2024-09-18T12:06:25.871756+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:25.871756+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:40.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:40.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19516, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867475+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:25.867475+0000', 'last_peered': '2024-09-18T12:06:25.867475+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:25.867475+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:40.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:40.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19471, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867373+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:06:25.867373+0000', 'last_peered': '2024-09-18T12:06:25.867373+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:06:25.867373+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:40.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:40.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26919, 'reported_epoch': 50, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:06:25.871651+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:06:25.871651+0000', 'last_peered': '2024-09-18T12:06:25.871651+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:06:25.871651+0000', 'last_undegraded': '2024-09-18T12:06:25.871651+0000', 'last_fullsized': '2024-09-18T12:06:25.871651+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:40.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:40.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19622, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.870033+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:25.870033+0000', 'last_peered': '2024-09-18T12:06:25.870033+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:25.870033+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:40.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:40.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 22056, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:34.394213+0000', 'last_change': '2024-09-18T12:06:25.584319+0000', 'last_active': '2024-09-18T12:06:34.394213+0000', 'last_peered': '2024-09-18T12:06:34.394213+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:06:34.394213+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2303, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1227, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:40.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:40.676 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:40.676 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16528/44 objects degraded (37563.636%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":800,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":51,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1747890176,"bytes_avail":288162402304,"bytes_total":289910292480,"degraded_objects":16528,"degraded_total":44,"degraded_ratio":375.63636363636363,"recovering_objects_per_sec":396,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":4760,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:40.689 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:40.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:40.987 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:40.989 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:40.999 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-18T12:06:39.304393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26066,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1706924,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281408596,"statfs":{"total":289910292480,"available":288162402304,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4760,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003083"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":22056,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:34.394213+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:34.394213+0000","last_peered":"2024-09-18T12:06:34.394213+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:34.394213+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1227,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26065,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051737,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41899999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.505},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.46200000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855499,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919428,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452412,"statfs":{"total":96636764160,"available":95695269888,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.308},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.38}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.45900000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:44.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:44.326 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:44.336 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-18T12:06:43.305823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26628,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1711084,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281404436,"statfs":{"total":289910292480,"available":288158142464,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4813,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003552"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":23180,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:39.394908+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:39.394908+0000","last_peered":"2024-09-18T12:06:39.394908+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:39.394908+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1789,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26627,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051737,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41899999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.505},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.46200000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855500,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923588,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448252,"statfs":{"total":96636764160,"available":95691010048,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.56399999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.57299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:44.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:44.722 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:44.733 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-18T12:06:43.305823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26628,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1711084,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281404436,"statfs":{"total":289910292480,"available":288158142464,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4813,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003552"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":23180,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:39.394908+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:39.394908+0000","last_peered":"2024-09-18T12:06:39.394908+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:39.394908+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1789,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26627,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051737,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41899999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.505},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.46200000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855500,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923588,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448252,"statfs":{"total":96636764160,"available":95691010048,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.56399999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":1.4159999999999999,"15min":1.1339999999999999},"min":{"1min":0.26100000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.56399999999999995,"5min":13.773,"15min":13.773},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.45600000000000002,"5min":1.1879999999999999,"15min":0.95699999999999996},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.59299999999999997,"5min":11.957000000000001,"15min":11.957000000000001},"last":0.57299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:44.736 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:44.736 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19619, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.871756+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:25.871756+0000', 'last_peered': '2024-09-18T12:06:25.871756+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:25.871756+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:44.736 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:44.736 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19516, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867475+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:25.867475+0000', 'last_peered': '2024-09-18T12:06:25.867475+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:25.867475+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:44.736 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:44.736 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19471, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867373+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:06:25.867373+0000', 'last_peered': '2024-09-18T12:06:25.867373+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:06:25.867373+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:44.737 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:44.737 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26919, 'reported_epoch': 50, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:06:25.871651+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:06:25.871651+0000', 'last_peered': '2024-09-18T12:06:25.871651+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:06:25.871651+0000', 'last_undegraded': '2024-09-18T12:06:25.871651+0000', 'last_fullsized': '2024-09-18T12:06:25.871651+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:44.737 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:44.737 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19622, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.870033+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:25.870033+0000', 'last_peered': '2024-09-18T12:06:25.870033+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:25.870033+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:44.737 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:44.738 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 23180, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:39.394908+0000', 'last_change': '2024-09-18T12:06:25.584319+0000', 'last_active': '2024-09-18T12:06:39.394908+0000', 'last_peered': '2024-09-18T12:06:39.394908+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:06:39.394908+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1741, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1789, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1737}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:44.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:45.118 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:45.118 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15966/44 objects degraded (36286.364%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":805,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":51,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1752150016,"bytes_avail":288158142464,"bytes_total":289910292480,"degraded_objects":15966,"degraded_total":44,"degraded_ratio":362.86363636363637,"recovering_objects_per_sec":400,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":4813,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:45.132 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:45.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:45.426 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:45.429 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:45.442 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2024-09-18T12:06:45.306559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27096,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1711084,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281404436,"statfs":{"total":289910292480,"available":288158142464,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1744,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1744,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003792"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":24116,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:44.397598+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:44.397598+0000","last_peered":"2024-09-18T12:06:44.397598+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:44.397598+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2257,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27095,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051738,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.59599999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.50800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855500,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923588,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448252,"statfs":{"total":96636764160,"available":95691010048,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.56399999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.48599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:48.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:48.751 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:48.753 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:48.763 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-09-18T12:06:47.306976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27096,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1715244,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281400276,"statfs":{"total":289910292480,"available":288153882624,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1030,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003691"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":24116,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:44.397598+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:44.397598+0000","last_peered":"2024-09-18T12:06:44.397598+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:44.397598+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2257,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27095,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051738,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.59599999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.50800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855501,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927748,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444092,"statfs":{"total":96636764160,"available":95686750208,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.42599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.496}]}]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.48599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:48.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:49.045 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:49.047 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:49.059 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-09-18T12:06:47.306976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27096,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1715244,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281400276,"statfs":{"total":289910292480,"available":288153882624,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1030,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003691"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":24116,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:44.397598+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:44.397598+0000","last_peered":"2024-09-18T12:06:44.397598+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:44.397598+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2257,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27095,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051738,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.59599999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.50800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855501,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927748,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444092,"statfs":{"total":96636764160,"available":95686750208,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.42599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.496}]}]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.48599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:49.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:49.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19619, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.871756+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:25.871756+0000', 'last_peered': '2024-09-18T12:06:25.871756+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:25.871756+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:49.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:49.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19516, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867475+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:25.867475+0000', 'last_peered': '2024-09-18T12:06:25.867475+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:25.867475+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:49.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:49.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19471, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867373+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:06:25.867373+0000', 'last_peered': '2024-09-18T12:06:25.867373+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:06:25.867373+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:49.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:49.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26919, 'reported_epoch': 50, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:06:25.871651+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:06:25.871651+0000', 'last_peered': '2024-09-18T12:06:25.871651+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:06:25.871651+0000', 'last_undegraded': '2024-09-18T12:06:25.871651+0000', 'last_fullsized': '2024-09-18T12:06:25.871651+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:49.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:49.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19622, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.870033+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:25.870033+0000', 'last_peered': '2024-09-18T12:06:25.870033+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:25.870033+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:49.062 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:49.062 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 24116, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:44.397598+0000', 'last_change': '2024-09-18T12:06:25.584319+0000', 'last_active': '2024-09-18T12:06:44.397598+0000', 'last_peered': '2024-09-18T12:06:44.397598+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:06:44.397598+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1273, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2257, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1269}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:49.062 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:49.422 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:49.422 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15498/44 objects degraded (35222.727%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":809,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":51,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1756409856,"bytes_avail":288153882624,"bytes_total":289910292480,"degraded_objects":15498,"degraded_total":44,"degraded_ratio":352.22727272727275,"recovering_objects_per_sec":85,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1030,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:49.438 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:49.438 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:49.745 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:49.748 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:49.760 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-18T12:06:49.307888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27096,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1715244,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281400276,"statfs":{"total":289910292480,"available":288153882624,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1030,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004102"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":24116,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:44.397598+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:44.397598+0000","last_peered":"2024-09-18T12:06:44.397598+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:44.397598+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2257,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27095,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051739,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.49199999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.627}]}]},{"osd":1,"up_from":35,"seq":150323855501,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927748,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444092,"statfs":{"total":96636764160,"available":95686750208,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.42599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.496}]}]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.48599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:52.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:53.072 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:53.074 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:53.084 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-18T12:06:51.308585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27634,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1715244,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281400276,"statfs":{"total":289910292480,"available":288153882624,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1568,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004191"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":25192,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:49.398403+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:49.398403+0000","last_peered":"2024-09-18T12:06:49.398403+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:49.398403+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2795,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27633,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051739,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.49199999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.627}]}]},{"osd":1,"up_from":35,"seq":150323855501,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927748,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444092,"statfs":{"total":96636764160,"available":95686750208,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.42599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.496}]}]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.57599999999999996}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.504},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:53.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:53.376 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:53.378 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:53.389 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-09-18T12:06:53.309179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27634,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1702764,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281412756,"statfs":{"total":289910292480,"available":288166662144,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1006,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004248"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":25192,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:49.398403+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:49.398403+0000","last_peered":"2024-09-18T12:06:49.398403+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:49.398403+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2795,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27633,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051739,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.49199999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.627}]}]},{"osd":1,"up_from":35,"seq":150323855502,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915268,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456572,"statfs":{"total":96636764160,"available":95699529728,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.435},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.34100000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.57599999999999996}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.504},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:53.390 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:53.391 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19619, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.871756+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:25.871756+0000', 'last_peered': '2024-09-18T12:06:25.871756+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:25.871756+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:53.391 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:53.391 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19516, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867475+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:25.867475+0000', 'last_peered': '2024-09-18T12:06:25.867475+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:25.867475+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:53.391 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:53.391 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19471, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867373+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:06:25.867373+0000', 'last_peered': '2024-09-18T12:06:25.867373+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:06:25.867373+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:53.391 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:53.391 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26919, 'reported_epoch': 50, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:06:25.871651+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:06:25.871651+0000', 'last_peered': '2024-09-18T12:06:25.871651+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:06:25.871651+0000', 'last_undegraded': '2024-09-18T12:06:25.871651+0000', 'last_fullsized': '2024-09-18T12:06:25.871651+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:53.391 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:53.391 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19622, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.870033+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:25.870033+0000', 'last_peered': '2024-09-18T12:06:25.870033+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:25.870033+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:53.391 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-18T12:06:53.392 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "43'19512", 'reported_seq': 25192, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:49.398403+0000', 'last_change': '2024-09-18T12:06:25.584319+0000', 'last_active': '2024-09-18T12:06:49.398403+0000', 'last_peered': '2024-09-18T12:06:49.398403+0000', 'last_clean': '2024-09-18T11:55:02.218861+0000', 'last_became_active': '2024-09-18T11:55:04.723243+0000', 'last_became_peered': '2024-09-18T11:55:04.723243+0000', 'last_unstale': '2024-09-18T12:06:49.398403+0000', 'last_undegraded': '2024-09-18T11:55:04.328211+0000', 'last_fullsized': '2024-09-18T11:55:04.248629+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2468, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 735, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19509, 'num_write_kb': 39007, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2795, 'num_bytes_recovered': 617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -731}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:53.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:53.764 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:53.764 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14960/44 objects degraded (34000.000%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":813,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":51,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1743630336,"bytes_avail":288166662144,"bytes_total":289910292480,"degraded_objects":14960,"degraded_total":44,"degraded_ratio":340,"recovering_objects_per_sec":83,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1006,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:53.781 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:53.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:54.050 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:54.053 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:54.063 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-09-18T12:06:53.309179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27634,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1702764,"kb_used_data":171464,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281412756,"statfs":{"total":289910292480,"available":288166662144,"internally_reserved":0,"allocated":175579136,"data_stored":174693080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1006,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004248"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19761,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-18T12:06:26.993788+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:26.993788+0000","last_peered":"2024-09-18T12:06:26.993788+0000","last_clean":"2024-09-18T12:06:26.993788+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:26.993788+0000","last_undegraded":"2024-09-18T12:06:26.993788+0000","last_fullsized":"2024-09-18T12:06:26.993788+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873571+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:25.873571+0000","last_peered":"2024-09-18T12:06:25.873571+0000","last_clean":"2024-09-18T12:06:25.873571+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:25.873571+0000","last_undegraded":"2024-09-18T12:06:25.873571+0000","last_fullsized":"2024-09-18T12:06:25.873571+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26545,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873866+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:25.873866+0000","last_peered":"2024-09-18T12:06:25.873866+0000","last_clean":"2024-09-18T12:06:25.873866+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:25.873866+0000","last_undegraded":"2024-09-18T12:06:25.873866+0000","last_fullsized":"2024-09-18T12:06:25.873866+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":83,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873774+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:25.873774+0000","last_peered":"2024-09-18T12:06:25.873774+0000","last_clean":"2024-09-18T12:06:25.873774+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:25.873774+0000","last_undegraded":"2024-09-18T12:06:25.873774+0000","last_fullsized":"2024-09-18T12:06:25.873774+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873847+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:25.873847+0000","last_peered":"2024-09-18T12:06:25.873847+0000","last_clean":"2024-09-18T12:06:25.873847+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:25.873847+0000","last_undegraded":"2024-09-18T12:06:25.873847+0000","last_fullsized":"2024-09-18T12:06:25.873847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26680,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874037+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:25.874037+0000","last_peered":"2024-09-18T12:06:25.874037+0000","last_clean":"2024-09-18T12:06:25.874037+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:25.874037+0000","last_undegraded":"2024-09-18T12:06:25.874037+0000","last_fullsized":"2024-09-18T12:06:25.874037+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19298,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.874489+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:25.874489+0000","last_peered":"2024-09-18T12:06:25.874489+0000","last_clean":"2024-09-18T12:06:25.874489+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:25.874489+0000","last_undegraded":"2024-09-18T12:06:25.874489+0000","last_fullsized":"2024-09-18T12:06:25.874489+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873953+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:25.873953+0000","last_peered":"2024-09-18T12:06:25.873953+0000","last_clean":"2024-09-18T12:06:25.873953+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:25.873953+0000","last_undegraded":"2024-09-18T12:06:25.873953+0000","last_fullsized":"2024-09-18T12:06:25.873953+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.873965+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:25.873965+0000","last_peered":"2024-09-18T12:06:25.873965+0000","last_clean":"2024-09-18T12:06:25.873965+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:25.873965+0000","last_undegraded":"2024-09-18T12:06:25.873965+0000","last_fullsized":"2024-09-18T12:06:25.873965+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":25192,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:49.398403+0000","last_change":"2024-09-18T12:06:25.584319+0000","last_active":"2024-09-18T12:06:49.398403+0000","last_peered":"2024-09-18T12:06:49.398403+0000","last_clean":"2024-09-18T11:55:02.218861+0000","last_became_active":"2024-09-18T11:55:04.723243+0000","last_became_peered":"2024-09-18T11:55:04.723243+0000","last_unstale":"2024-09-18T12:06:49.398403+0000","last_undegraded":"2024-09-18T11:55:04.328211+0000","last_fullsized":"2024-09-18T11:55:04.248629+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2795,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14960,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27633,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174292992,"data_stored":174265160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051739,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.49199999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.627}]}]},{"osd":1,"up_from":35,"seq":150323855502,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915268,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456572,"statfs":{"total":96636764160,"available":95699529728,"internally_reserved":0,"allocated":454656,"data_stored":146500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.435},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.34100000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.57599999999999996}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.504},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:57.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:57.375 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:57.377 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:57.388 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-18T12:06:57.310689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28369,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1711096,"kb_used_data":171472,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281404424,"statfs":{"total":289910292480,"available":288158130176,"internally_reserved":0,"allocated":175587328,"data_stored":174697461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1273,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002800"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28368,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174284800,"data_stored":174263926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051740,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.37},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.47099999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.504}]}]},{"osd":1,"up_from":35,"seq":150323855503,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923600,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448240,"statfs":{"total":96636764160,"available":95690997760,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.435}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.52000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.35899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.621}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:57.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:57.707 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:57.709 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:57.718 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-18T12:06:57.310689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28369,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1711096,"kb_used_data":171472,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281404424,"statfs":{"total":289910292480,"available":288158130176,"internally_reserved":0,"allocated":175587328,"data_stored":174697461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1273,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002800"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28368,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174284800,"data_stored":174263926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051740,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.37},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.47099999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.504}]}]},{"osd":1,"up_from":35,"seq":150323855503,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923600,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448240,"statfs":{"total":96636764160,"available":95690997760,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.435}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.52000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.35899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.621}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:06:57.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:06:57.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19619, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.871756+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:25.871756+0000', 'last_peered': '2024-09-18T12:06:25.871756+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:25.871756+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:57.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:06:57.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19516, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867475+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:25.867475+0000', 'last_peered': '2024-09-18T12:06:25.867475+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:25.867475+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:57.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:06:57.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 19471, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.867373+0000', 'last_change': '2024-09-18T11:55:04.772924+0000', 'last_active': '2024-09-18T12:06:25.867373+0000', 'last_peered': '2024-09-18T12:06:25.867373+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:06:25.867373+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:57.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-18T12:06:57.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "43'19498", 'reported_seq': 26919, 'reported_epoch': 50, 'state': 'active+recovery_wait', 'last_fresh': '2024-09-18T12:06:25.871651+0000', 'last_change': '2024-09-18T12:01:44.625806+0000', 'last_active': '2024-09-18T12:06:25.871651+0000', 'last_peered': '2024-09-18T12:06:25.871651+0000', 'last_clean': '2024-09-18T11:55:02.228397+0000', 'last_became_active': '2024-09-18T12:01:44.514394+0000', 'last_became_peered': '2024-09-18T12:01:44.514394+0000', 'last_unstale': '2024-09-18T12:06:25.871651+0000', 'last_undegraded': '2024-09-18T12:06:25.871651+0000', 'last_fullsized': '2024-09-18T12:06:25.871651+0000', 'mapping_epoch': 42, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "34'3593", 'last_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:55:02.143655+0000', 'objects_scrubbed': 3514, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 3, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000', 'scrub_duration': 2.374930532, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15978, 'num_write_kb': 24928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3596, 'num_bytes_recovered': 7106560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '0', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T12:06:57.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:06:57.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19622, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:25.870033+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:25.870033+0000', 'last_peered': '2024-09-18T12:06:25.870033+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:25.870033+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:06:57.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:06:58.088 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:58.088 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14225/44 objects degraded (32329.545%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":818,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovery_wait","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1752162304,"bytes_avail":288158130176,"bytes_total":289910292480,"degraded_objects":14225,"degraded_total":44,"degraded_ratio":323.29545454545456,"recovering_objects_per_sec":159,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1273,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:06:58.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:06:58.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:06:58.412 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:06:58.416 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:06:58.432 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-18T12:06:57.310689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28369,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1711096,"kb_used_data":171472,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281404424,"statfs":{"total":289910292480,"available":288158130176,"internally_reserved":0,"allocated":175587328,"data_stored":174697461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1273,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002800"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19619,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.871756+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:25.871756+0000","last_peered":"2024-09-18T12:06:25.871756+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:25.871756+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19602,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872840+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:25.872840+0000","last_peered":"2024-09-18T12:06:25.872840+0000","last_clean":"2024-09-18T12:06:25.872840+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:25.872840+0000","last_undegraded":"2024-09-18T12:06:25.872840+0000","last_fullsized":"2024-09-18T12:06:25.872840+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19301,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872333+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:25.872333+0000","last_peered":"2024-09-18T12:06:25.872333+0000","last_clean":"2024-09-18T12:06:25.872333+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:25.872333+0000","last_undegraded":"2024-09-18T12:06:25.872333+0000","last_fullsized":"2024-09-18T12:06:25.872333+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19650,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866734+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:25.866734+0000","last_peered":"2024-09-18T12:06:25.866734+0000","last_clean":"2024-09-18T12:06:25.866734+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:25.866734+0000","last_undegraded":"2024-09-18T12:06:25.866734+0000","last_fullsized":"2024-09-18T12:06:25.866734+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19516,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867475+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:25.867475+0000","last_peered":"2024-09-18T12:06:25.867475+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:25.867475+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19471,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.867373+0000","last_change":"2024-09-18T11:55:04.772924+0000","last_active":"2024-09-18T12:06:25.867373+0000","last_peered":"2024-09-18T12:06:25.867373+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:25.867373+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19839,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866798+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:25.866798+0000","last_peered":"2024-09-18T12:06:25.866798+0000","last_clean":"2024-09-18T12:06:25.866798+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:25.866798+0000","last_undegraded":"2024-09-18T12:06:25.866798+0000","last_fullsized":"2024-09-18T12:06:25.866798+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871680+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:25.871680+0000","last_peered":"2024-09-18T12:06:25.871680+0000","last_clean":"2024-09-18T12:06:25.871680+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:25.871680+0000","last_undegraded":"2024-09-18T12:06:25.871680+0000","last_fullsized":"2024-09-18T12:06:25.871680+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26615,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.867521+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:25.867521+0000","last_peered":"2024-09-18T12:06:25.867521+0000","last_clean":"2024-09-18T12:06:25.867521+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:25.867521+0000","last_undegraded":"2024-09-18T12:06:25.867521+0000","last_fullsized":"2024-09-18T12:06:25.867521+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":103,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.866916+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:25.866916+0000","last_peered":"2024-09-18T12:06:25.866916+0000","last_clean":"2024-09-18T12:06:25.866916+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:25.866916+0000","last_undegraded":"2024-09-18T12:06:25.866916+0000","last_fullsized":"2024-09-18T12:06:25.866916+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26646,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.871906+0000","last_change":"2024-09-18T12:05:18.557713+0000","last_active":"2024-09-18T12:06:25.871906+0000","last_peered":"2024-09-18T12:06:25.871906+0000","last_clean":"2024-09-18T12:06:25.871906+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:25.871906+0000","last_undegraded":"2024-09-18T12:06:25.871906+0000","last_fullsized":"2024-09-18T12:06:25.871906+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26919,"reported_epoch":50,"state":"active+recovery_wait","last_fresh":"2024-09-18T12:06:25.871651+0000","last_change":"2024-09-18T12:01:44.625806+0000","last_active":"2024-09-18T12:06:25.871651+0000","last_peered":"2024-09-18T12:06:25.871651+0000","last_clean":"2024-09-18T11:55:02.228397+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:25.871651+0000","last_undegraded":"2024-09-18T12:06:25.871651+0000","last_fullsized":"2024-09-18T12:06:25.871651+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19622,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:25.870033+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:25.870033+0000","last_peered":"2024-09-18T12:06:25.870033+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:25.870033+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-18T12:06:25.872433+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:25.872433+0000","last_peered":"2024-09-18T12:06:25.872433+0000","last_clean":"2024-09-18T12:06:25.872433+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:25.872433+0000","last_undegraded":"2024-09-18T12:06:25.872433+0000","last_fullsized":"2024-09-18T12:06:25.872433+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28368,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174284800,"data_stored":174263926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051740,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420168,"kb_used_data":113976,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93951672,"statfs":{"total":96636764160,"available":96206512128,"internally_reserved":0,"allocated":116711424,"data_stored":116424285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.37},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.47099999999999997}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.504}]}]},{"osd":1,"up_from":35,"seq":150323855503,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923600,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448240,"statfs":{"total":96636764160,"available":95690997760,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.435}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.52000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367328,"kb_used_data":57044,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004512,"statfs":{"total":96636764160,"available":96260620288,"internally_reserved":0,"allocated":58413056,"data_stored":58122295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.35899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.621}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":116285440,"data_stored":116281961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:01.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:01.728 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:01.731 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:01.741 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-18T12:07:01.311619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28973,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1709340,"kb_used_data":169716,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281406180,"statfs":{"total":289910292480,"available":288159928320,"internally_reserved":0,"allocated":173789184,"data_stored":172881856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1339,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002440"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":20682,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:59.394935+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:06:59.394935+0000","last_peered":"2024-09-18T12:06:59.394935+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:59.394935+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":607,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26651,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.450227+0000","last_change":"2024-09-18T12:06:56.450227+0000","last_active":"2024-09-18T12:06:56.450227+0000","last_peered":"2024-09-18T12:06:56.450227+0000","last_clean":"2024-09-18T12:06:56.450227+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:56.450227+0000","last_undegraded":"2024-09-18T12:06:56.450227+0000","last_fullsized":"2024-09-18T12:06:56.450227+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28972,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172089344,"data_stored":172068470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051741,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418392,"kb_used_data":112204,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93953448,"statfs":{"total":96636764160,"available":96208330752,"internally_reserved":0,"allocated":114896896,"data_stored":114603065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.36499999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.41099999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855503,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923600,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448240,"statfs":{"total":96636764160,"available":95690997760,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.435}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.52000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.255}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.309},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.33800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114089984,"data_stored":114086505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:01.742 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:02.048 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:02.051 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:02.063 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-18T12:07:01.311619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28973,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1709340,"kb_used_data":169716,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281406180,"statfs":{"total":289910292480,"available":288159928320,"internally_reserved":0,"allocated":173789184,"data_stored":172881856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1339,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002440"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":20682,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:59.394935+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:06:59.394935+0000","last_peered":"2024-09-18T12:06:59.394935+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:59.394935+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":607,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26651,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.450227+0000","last_change":"2024-09-18T12:06:56.450227+0000","last_active":"2024-09-18T12:06:56.450227+0000","last_peered":"2024-09-18T12:06:56.450227+0000","last_clean":"2024-09-18T12:06:56.450227+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:56.450227+0000","last_undegraded":"2024-09-18T12:06:56.450227+0000","last_fullsized":"2024-09-18T12:06:56.450227+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28972,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172089344,"data_stored":172068470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051741,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418392,"kb_used_data":112204,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93953448,"statfs":{"total":96636764160,"available":96208330752,"internally_reserved":0,"allocated":114896896,"data_stored":114603065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.36499999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.41099999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855503,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923600,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448240,"statfs":{"total":96636764160,"available":95690997760,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.435}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.52000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.255}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.309},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.33800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114089984,"data_stored":114086505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:02.064 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:02.064 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19621, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099497+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:55.099497+0000', 'last_peered': '2024-09-18T12:06:55.099497+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:55.099497+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:02.065 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:02.065 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19518, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.097395+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:55.097395+0000', 'last_peered': '2024-09-18T12:06:55.097395+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:55.097395+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:02.065 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:07:02.065 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 20682, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:59.394935+0000', 'last_change': '2024-09-18T12:06:54.866165+0000', 'last_active': '2024-09-18T12:06:59.394935+0000', 'last_peered': '2024-09-18T12:06:59.394935+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:06:59.394935+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2957, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 607, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2955}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:02.065 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:02.065 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19624, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099413+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:55.099413+0000', 'last_peered': '2024-09-18T12:06:55.099413+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:55.099413+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:02.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:02.450 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:02.450 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13621/44 objects degraded (30956.818%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":822,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1750364160,"bytes_avail":288159928320,"bytes_total":289910292480,"degraded_objects":13621,"degraded_total":44,"degraded_ratio":309.56818181818181,"recovering_objects_per_sec":167,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1339,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:02.465 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:02.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:02.762 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:02.765 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:02.778 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-18T12:07:01.311619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28973,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1709340,"kb_used_data":169716,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281406180,"statfs":{"total":289910292480,"available":288159928320,"internally_reserved":0,"allocated":173789184,"data_stored":172881856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1339,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002440"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":20682,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:59.394935+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:06:59.394935+0000","last_peered":"2024-09-18T12:06:59.394935+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:06:59.394935+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":607,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26651,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.450227+0000","last_change":"2024-09-18T12:06:56.450227+0000","last_active":"2024-09-18T12:06:56.450227+0000","last_peered":"2024-09-18T12:06:56.450227+0000","last_clean":"2024-09-18T12:06:56.450227+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:06:56.450227+0000","last_undegraded":"2024-09-18T12:06:56.450227+0000","last_fullsized":"2024-09-18T12:06:56.450227+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28972,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172089344,"data_stored":172068470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051741,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418392,"kb_used_data":112204,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93953448,"statfs":{"total":96636764160,"available":96208330752,"internally_reserved":0,"allocated":114896896,"data_stored":114603065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.36499999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.41099999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855503,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923600,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448240,"statfs":{"total":96636764160,"available":95690997760,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.435}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.52000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.255}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.309},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.33800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114089984,"data_stored":114086505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:05.780 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:06.085 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:06.087 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:06.099 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-18T12:07:05.312770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29674,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1711092,"kb_used_data":167316,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281404428,"statfs":{"total":289910292480,"available":288158134272,"internally_reserved":0,"allocated":171331584,"data_stored":170424256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1378,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002751"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":22084,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:04.401817+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:04.401817+0000","last_peered":"2024-09-18T12:07:04.401817+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:04.401817+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1308,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26657,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:03.441886+0000","last_change":"2024-09-18T12:07:03.441886+0000","last_active":"2024-09-18T12:07:03.441886+0000","last_peered":"2024-09-18T12:07:03.441886+0000","last_clean":"2024-09-18T12:07:03.441886+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:03.441886+0000","last_undegraded":"2024-09-18T12:07:03.441886+0000","last_fullsized":"2024-09-18T12:07:03.441886+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29673,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":169631744,"data_stored":169610870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051742,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415988,"kb_used_data":109804,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93955852,"statfs":{"total":96636764160,"available":96210792448,"internally_reserved":0,"allocated":112439296,"data_stored":112145465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.25800000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.28999999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855504,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444084,"statfs":{"total":96636764160,"available":95686742016,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.32000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.36499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":111632384,"data_stored":111628905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:06.100 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:06.396 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:06.410 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-18T12:07:05.312770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29674,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1711092,"kb_used_data":167316,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281404428,"statfs":{"total":289910292480,"available":288158134272,"internally_reserved":0,"allocated":171331584,"data_stored":170424256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1378,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002751"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":22084,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:04.401817+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:04.401817+0000","last_peered":"2024-09-18T12:07:04.401817+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:04.401817+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1308,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26657,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:03.441886+0000","last_change":"2024-09-18T12:07:03.441886+0000","last_active":"2024-09-18T12:07:03.441886+0000","last_peered":"2024-09-18T12:07:03.441886+0000","last_clean":"2024-09-18T12:07:03.441886+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:03.441886+0000","last_undegraded":"2024-09-18T12:07:03.441886+0000","last_fullsized":"2024-09-18T12:07:03.441886+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29673,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":169631744,"data_stored":169610870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051742,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415988,"kb_used_data":109804,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93955852,"statfs":{"total":96636764160,"available":96210792448,"internally_reserved":0,"allocated":112439296,"data_stored":112145465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.25800000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.28999999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855504,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444084,"statfs":{"total":96636764160,"available":95686742016,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.32000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.36499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":111632384,"data_stored":111628905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:06.411 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:06.411 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19621, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099497+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:55.099497+0000', 'last_peered': '2024-09-18T12:06:55.099497+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:55.099497+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:06.412 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:06.412 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19518, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.097395+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:55.097395+0000', 'last_peered': '2024-09-18T12:06:55.097395+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:55.097395+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:06.412 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:07:06.412 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 22084, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:04.401817+0000', 'last_change': '2024-09-18T12:06:54.866165+0000', 'last_active': '2024-09-18T12:07:04.401817+0000', 'last_peered': '2024-09-18T12:07:04.401817+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:07:04.401817+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2256, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1308, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2254}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:06.412 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:06.412 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19624, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099413+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:55.099413+0000', 'last_peered': '2024-09-18T12:06:55.099413+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:55.099413+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:06.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:06.780 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:06.781 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12920/44 objects degraded (29363.636%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":826,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1752158208,"bytes_avail":288158134272,"bytes_total":289910292480,"degraded_objects":12920,"degraded_total":44,"degraded_ratio":293.63636363636363,"recovering_objects_per_sec":137,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1378,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:06.795 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:06.795 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:07.100 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:07.102 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:07.112 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-18T12:07:05.312770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29674,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1711092,"kb_used_data":167316,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281404428,"statfs":{"total":289910292480,"available":288158134272,"internally_reserved":0,"allocated":171331584,"data_stored":170424256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1378,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002751"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":22084,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:04.401817+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:04.401817+0000","last_peered":"2024-09-18T12:07:04.401817+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:04.401817+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1308,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26657,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:03.441886+0000","last_change":"2024-09-18T12:07:03.441886+0000","last_active":"2024-09-18T12:07:03.441886+0000","last_peered":"2024-09-18T12:07:03.441886+0000","last_clean":"2024-09-18T12:07:03.441886+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:03.441886+0000","last_undegraded":"2024-09-18T12:07:03.441886+0000","last_fullsized":"2024-09-18T12:07:03.441886+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29673,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":169631744,"data_stored":169610870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051742,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":415988,"kb_used_data":109804,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93955852,"statfs":{"total":96636764160,"available":96210792448,"internally_reserved":0,"allocated":112439296,"data_stored":112145465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.25800000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.28999999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855504,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444084,"statfs":{"total":96636764160,"available":95686742016,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.32000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.36499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":111632384,"data_stored":111628905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:10.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:10.424 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:10.426 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:10.436 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-18T12:07:09.313902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29674,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1696212,"kb_used_data":164916,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281419308,"statfs":{"total":289910292480,"available":288173371392,"internally_reserved":0,"allocated":168873984,"data_stored":167966656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1305,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003213"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":22084,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:04.401817+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:04.401817+0000","last_peered":"2024-09-18T12:07:04.401817+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:04.401817+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1308,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26657,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:03.441886+0000","last_change":"2024-09-18T12:07:03.441886+0000","last_active":"2024-09-18T12:07:03.441886+0000","last_peered":"2024-09-18T12:07:03.441886+0000","last_clean":"2024-09-18T12:07:03.441886+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:03.441886+0000","last_undegraded":"2024-09-18T12:07:03.441886+0000","last_fullsized":"2024-09-18T12:07:03.441886+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29673,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167174144,"data_stored":167153270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051743,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413588,"kb_used_data":107404,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93958252,"statfs":{"total":96636764160,"available":96213250048,"internally_reserved":0,"allocated":109981696,"data_stored":109687865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.26700000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855505,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915276,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456564,"statfs":{"total":96636764160,"available":95699521536,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.246}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.309}]}]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.36499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":109174784,"data_stored":109171305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:10.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:10.738 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:10.748 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-18T12:07:09.313902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29674,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1696212,"kb_used_data":164916,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281419308,"statfs":{"total":289910292480,"available":288173371392,"internally_reserved":0,"allocated":168873984,"data_stored":167966656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1305,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003213"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":22084,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:04.401817+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:04.401817+0000","last_peered":"2024-09-18T12:07:04.401817+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:04.401817+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1308,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26657,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:03.441886+0000","last_change":"2024-09-18T12:07:03.441886+0000","last_active":"2024-09-18T12:07:03.441886+0000","last_peered":"2024-09-18T12:07:03.441886+0000","last_clean":"2024-09-18T12:07:03.441886+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:03.441886+0000","last_undegraded":"2024-09-18T12:07:03.441886+0000","last_fullsized":"2024-09-18T12:07:03.441886+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29673,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167174144,"data_stored":167153270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051743,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413588,"kb_used_data":107404,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93958252,"statfs":{"total":96636764160,"available":96213250048,"internally_reserved":0,"allocated":109981696,"data_stored":109687865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.26700000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855505,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915276,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456564,"statfs":{"total":96636764160,"available":95699521536,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.246}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.309}]}]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.36499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":109174784,"data_stored":109171305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:10.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:10.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19621, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099497+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:55.099497+0000', 'last_peered': '2024-09-18T12:06:55.099497+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:55.099497+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:10.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:10.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19518, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.097395+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:55.097395+0000', 'last_peered': '2024-09-18T12:06:55.097395+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:55.097395+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:10.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:07:10.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 22084, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:04.401817+0000', 'last_change': '2024-09-18T12:06:54.866165+0000', 'last_active': '2024-09-18T12:07:04.401817+0000', 'last_peered': '2024-09-18T12:07:04.401817+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:07:04.401817+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2256, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1308, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2254}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:10.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:10.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19624, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099413+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:55.099413+0000', 'last_peered': '2024-09-18T12:06:55.099413+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:55.099413+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:10.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:11.096 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:11.096 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12920/44 objects degraded (29363.636%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":831,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1736921088,"bytes_avail":288173371392,"bytes_total":289910292480,"degraded_objects":12920,"degraded_total":44,"degraded_ratio":293.63636363636363,"recovering_objects_per_sec":108,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1305,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:11.116 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:11.116 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:11.404 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:11.406 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:11.416 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-18T12:07:11.314773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30257,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1696212,"kb_used_data":164916,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281419308,"statfs":{"total":289910292480,"available":288173371392,"internally_reserved":0,"allocated":168873984,"data_stored":167966656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1888,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1888,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003644"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":23250,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:09.407181+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:09.407181+0000","last_peered":"2024-09-18T12:07:09.407181+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:09.407181+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1891,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26657,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:03.441886+0000","last_change":"2024-09-18T12:07:03.441886+0000","last_active":"2024-09-18T12:07:03.441886+0000","last_peered":"2024-09-18T12:07:03.441886+0000","last_clean":"2024-09-18T12:07:03.441886+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:03.441886+0000","last_undegraded":"2024-09-18T12:07:03.441886+0000","last_fullsized":"2024-09-18T12:07:03.441886+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30256,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167174144,"data_stored":167153270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051743,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413588,"kb_used_data":107404,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93958252,"statfs":{"total":96636764160,"available":96213250048,"internally_reserved":0,"allocated":109981696,"data_stored":109687865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.26700000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855505,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915276,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456564,"statfs":{"total":96636764160,"available":95699521536,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.246}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.309}]}]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.52500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":109174784,"data_stored":109171305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:14.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:14.731 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:14.734 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:14.749 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-18T12:07:13.315472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30257,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1700372,"kb_used_data":164916,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281415148,"statfs":{"total":289910292480,"available":288169111552,"internally_reserved":0,"allocated":168873984,"data_stored":167966656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1284,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003852"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":23250,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:09.407181+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:09.407181+0000","last_peered":"2024-09-18T12:07:09.407181+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:09.407181+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1891,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26657,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:03.441886+0000","last_change":"2024-09-18T12:07:03.441886+0000","last_active":"2024-09-18T12:07:03.441886+0000","last_peered":"2024-09-18T12:07:03.441886+0000","last_clean":"2024-09-18T12:07:03.441886+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:03.441886+0000","last_undegraded":"2024-09-18T12:07:03.441886+0000","last_fullsized":"2024-09-18T12:07:03.441886+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30256,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167174144,"data_stored":167153270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051743,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413588,"kb_used_data":107404,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93958252,"statfs":{"total":96636764160,"available":96213250048,"internally_reserved":0,"allocated":109981696,"data_stored":109687865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.26700000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855506,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919436,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452404,"statfs":{"total":96636764160,"available":95695261696,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.52300000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.52500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":109174784,"data_stored":109171305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:14.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:15.029 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:15.031 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:15.041 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-18T12:07:13.315472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30257,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1700372,"kb_used_data":164916,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281415148,"statfs":{"total":289910292480,"available":288169111552,"internally_reserved":0,"allocated":168873984,"data_stored":167966656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1284,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003852"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":23250,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:09.407181+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:09.407181+0000","last_peered":"2024-09-18T12:07:09.407181+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:09.407181+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1891,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26657,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:03.441886+0000","last_change":"2024-09-18T12:07:03.441886+0000","last_active":"2024-09-18T12:07:03.441886+0000","last_peered":"2024-09-18T12:07:03.441886+0000","last_clean":"2024-09-18T12:07:03.441886+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:03.441886+0000","last_undegraded":"2024-09-18T12:07:03.441886+0000","last_fullsized":"2024-09-18T12:07:03.441886+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30256,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167174144,"data_stored":167153270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051743,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413588,"kb_used_data":107404,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93958252,"statfs":{"total":96636764160,"available":96213250048,"internally_reserved":0,"allocated":109981696,"data_stored":109687865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.26700000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855506,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919436,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452404,"statfs":{"total":96636764160,"available":95695261696,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.52300000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.52500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":109174784,"data_stored":109171305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:15.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:15.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19621, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099497+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:55.099497+0000', 'last_peered': '2024-09-18T12:06:55.099497+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:55.099497+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:15.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:15.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19518, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.097395+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:55.097395+0000', 'last_peered': '2024-09-18T12:06:55.097395+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:55.097395+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:15.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:07:15.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 23250, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:09.407181+0000', 'last_change': '2024-09-18T12:06:54.866165+0000', 'last_active': '2024-09-18T12:07:09.407181+0000', 'last_peered': '2024-09-18T12:07:09.407181+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:07:09.407181+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1673, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1891, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1671}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:15.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:15.043 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19624, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099413+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:55.099413+0000', 'last_peered': '2024-09-18T12:06:55.099413+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:55.099413+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:15.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:15.381 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:15.382 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12337/44 objects degraded (28038.636%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":835,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1741180928,"bytes_avail":288169111552,"bytes_total":289910292480,"degraded_objects":12337,"degraded_total":44,"degraded_ratio":280.38636363636363,"recovering_objects_per_sec":106,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1284,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:15.395 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:15.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:15.677 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:15.681 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:15.693 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-09-18T12:07:15.316030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30729,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1697972,"kb_used_data":162516,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281417548,"statfs":{"total":289910292480,"available":288171569152,"internally_reserved":0,"allocated":166416384,"data_stored":165509056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004005"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":24194,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:14.402290+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:14.402290+0000","last_peered":"2024-09-18T12:07:14.402290+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:14.402290+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2363,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1199}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26666,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:13.507122+0000","last_change":"2024-09-18T12:07:13.507122+0000","last_active":"2024-09-18T12:07:13.507122+0000","last_peered":"2024-09-18T12:07:13.507122+0000","last_clean":"2024-09-18T12:07:13.507122+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:13.507122+0000","last_undegraded":"2024-09-18T12:07:13.507122+0000","last_fullsized":"2024-09-18T12:07:13.507122+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30728,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":164716544,"data_stored":164695670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051744,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411188,"kb_used_data":105004,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93960652,"statfs":{"total":96636764160,"available":96215707648,"internally_reserved":0,"allocated":107524096,"data_stored":107230265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.316}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.28299999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855506,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919436,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452404,"statfs":{"total":96636764160,"available":95695261696,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.52300000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.497},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.70599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106717184,"data_stored":106713705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:18.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:19.006 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:19.008 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:19.018 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-09-18T12:07:17.316797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30729,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1702132,"kb_used_data":162516,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281413388,"statfs":{"total":289910292480,"available":288167309312,"internally_reserved":0,"allocated":166416384,"data_stored":165509056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004027"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":24194,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:14.402290+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:14.402290+0000","last_peered":"2024-09-18T12:07:14.402290+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:14.402290+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2363,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1199}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26666,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:13.507122+0000","last_change":"2024-09-18T12:07:13.507122+0000","last_active":"2024-09-18T12:07:13.507122+0000","last_peered":"2024-09-18T12:07:13.507122+0000","last_clean":"2024-09-18T12:07:13.507122+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:13.507122+0000","last_undegraded":"2024-09-18T12:07:13.507122+0000","last_fullsized":"2024-09-18T12:07:13.507122+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30728,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":164716544,"data_stored":164695670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051744,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411188,"kb_used_data":105004,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93960652,"statfs":{"total":96636764160,"available":96215707648,"internally_reserved":0,"allocated":107524096,"data_stored":107230265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.316}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.28299999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855507,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923596,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448244,"statfs":{"total":96636764160,"available":95691001856,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.49399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.47299999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.497},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.70599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106717184,"data_stored":106713705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:19.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:19.346 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:19.349 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:19.363 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-18T12:07:19.317479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30729,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1699732,"kb_used_data":160116,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281415788,"statfs":{"total":289910292480,"available":288169766912,"internally_reserved":0,"allocated":163958784,"data_stored":163051456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004230"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":24194,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:14.402290+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:14.402290+0000","last_peered":"2024-09-18T12:07:14.402290+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:14.402290+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2363,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1199}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26669,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:14.505218+0000","last_change":"2024-09-18T12:07:14.505218+0000","last_active":"2024-09-18T12:07:14.505218+0000","last_peered":"2024-09-18T12:07:14.505218+0000","last_clean":"2024-09-18T12:07:14.505218+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:14.505218+0000","last_undegraded":"2024-09-18T12:07:14.505218+0000","last_fullsized":"2024-09-18T12:07:14.505218+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30728,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":162258944,"data_stored":162238070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051745,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408788,"kb_used_data":102604,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93963052,"statfs":{"total":96636764160,"available":96218165248,"internally_reserved":0,"allocated":105066496,"data_stored":104772665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.36099999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855507,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923596,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448244,"statfs":{"total":96636764160,"available":95691001856,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.49399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.47299999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.497},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.70599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104259584,"data_stored":104256105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:19.364 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:19.364 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19621, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099497+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:55.099497+0000', 'last_peered': '2024-09-18T12:06:55.099497+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:55.099497+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:19.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:19.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19518, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.097395+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:55.097395+0000', 'last_peered': '2024-09-18T12:06:55.097395+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:55.097395+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:19.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:07:19.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 24194, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:14.402290+0000', 'last_change': '2024-09-18T12:06:54.866165+0000', 'last_active': '2024-09-18T12:07:14.402290+0000', 'last_peered': '2024-09-18T12:07:14.402290+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:07:14.402290+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1201, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2363, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1199}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:19.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:19.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19624, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099413+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:55.099413+0000', 'last_peered': '2024-09-18T12:06:55.099413+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:55.099413+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:19.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:19.709 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:19.709 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11865/44 objects degraded (26965.909%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":839,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1740525568,"bytes_avail":288169766912,"bytes_total":289910292480,"degraded_objects":11865,"degraded_total":44,"degraded_ratio":269.65909090909093,"recovering_objects_per_sec":87,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1055,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:19.723 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:19.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:20.014 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:20.015 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:20.026 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-18T12:07:19.317479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30729,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1699732,"kb_used_data":160116,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281415788,"statfs":{"total":289910292480,"available":288169766912,"internally_reserved":0,"allocated":163958784,"data_stored":163051456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004230"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":24194,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:14.402290+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:14.402290+0000","last_peered":"2024-09-18T12:07:14.402290+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:14.402290+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2363,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1199}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26669,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:14.505218+0000","last_change":"2024-09-18T12:07:14.505218+0000","last_active":"2024-09-18T12:07:14.505218+0000","last_peered":"2024-09-18T12:07:14.505218+0000","last_clean":"2024-09-18T12:07:14.505218+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:14.505218+0000","last_undegraded":"2024-09-18T12:07:14.505218+0000","last_fullsized":"2024-09-18T12:07:14.505218+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30728,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":162258944,"data_stored":162238070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051745,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408788,"kb_used_data":102604,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93963052,"statfs":{"total":96636764160,"available":96218165248,"internally_reserved":0,"allocated":105066496,"data_stored":104772665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.36099999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855507,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923596,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448244,"statfs":{"total":96636764160,"available":95691001856,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.49399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.47299999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.497},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.70599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104259584,"data_stored":104256105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:23.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:23.346 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:23.348 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:23.358 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-18T12:07:23.318652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31274,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1703892,"kb_used_data":160116,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281411628,"statfs":{"total":289910292480,"available":288165507072,"internally_reserved":0,"allocated":163958784,"data_stored":163051456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003879"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":25284,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:19.407803+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:19.407803+0000","last_peered":"2024-09-18T12:07:19.407803+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:19.407803+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26669,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:14.505218+0000","last_change":"2024-09-18T12:07:14.505218+0000","last_active":"2024-09-18T12:07:14.505218+0000","last_peered":"2024-09-18T12:07:14.505218+0000","last_clean":"2024-09-18T12:07:14.505218+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:14.505218+0000","last_undegraded":"2024-09-18T12:07:14.505218+0000","last_fullsized":"2024-09-18T12:07:14.505218+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31273,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":162258944,"data_stored":162238070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051745,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408788,"kb_used_data":102604,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93963052,"statfs":{"total":96636764160,"available":96218165248,"internally_reserved":0,"allocated":105066496,"data_stored":104772665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.36099999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855508,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444084,"statfs":{"total":96636764160,"available":95686742016,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.38300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.41099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.69099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104259584,"data_stored":104256105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:23.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:23.647 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:23.650 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:23.661 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-18T12:07:23.318652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31274,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1703892,"kb_used_data":160116,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281411628,"statfs":{"total":289910292480,"available":288165507072,"internally_reserved":0,"allocated":163958784,"data_stored":163051456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003879"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":25284,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:19.407803+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:19.407803+0000","last_peered":"2024-09-18T12:07:19.407803+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:19.407803+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26669,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:14.505218+0000","last_change":"2024-09-18T12:07:14.505218+0000","last_active":"2024-09-18T12:07:14.505218+0000","last_peered":"2024-09-18T12:07:14.505218+0000","last_clean":"2024-09-18T12:07:14.505218+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:14.505218+0000","last_undegraded":"2024-09-18T12:07:14.505218+0000","last_fullsized":"2024-09-18T12:07:14.505218+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31273,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":162258944,"data_stored":162238070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051745,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408788,"kb_used_data":102604,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93963052,"statfs":{"total":96636764160,"available":96218165248,"internally_reserved":0,"allocated":105066496,"data_stored":104772665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.36099999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855508,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444084,"statfs":{"total":96636764160,"available":95686742016,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.38300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.41099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.69099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104259584,"data_stored":104256105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:23.663 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:23.663 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19621, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099497+0000', 'last_change': '2024-09-18T11:55:04.779429+0000', 'last_active': '2024-09-18T12:06:55.099497+0000', 'last_peered': '2024-09-18T12:06:55.099497+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:06:55.099497+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3599, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3599}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:23.663 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:23.663 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19518, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.097395+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:55.097395+0000', 'last_peered': '2024-09-18T12:06:55.097395+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:55.097395+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:23.664 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-18T12:07:23.664 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "43'19372", 'reported_seq': 25284, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:19.407803+0000', 'last_change': '2024-09-18T12:06:54.866165+0000', 'last_active': '2024-09-18T12:07:19.407803+0000', 'last_peered': '2024-09-18T12:07:19.407803+0000', 'last_clean': '2024-09-18T11:55:02.213764+0000', 'last_became_active': '2024-09-18T11:55:04.764665+0000', 'last_became_peered': '2024-09-18T11:55:04.764665+0000', 'last_unstale': '2024-09-18T12:07:19.407803+0000', 'last_undegraded': '2024-09-18T11:55:04.354089+0000', 'last_fullsized': '2024-09-18T11:55:04.248775+0000', 'mapping_epoch': 36, 'log_start': "39'9300", 'ondisk_log_start': "39'9300", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 656, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19370, 'num_write_kb': 38736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2908, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -654}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:23.664 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:23.664 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19624, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099413+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:55.099413+0000', 'last_peered': '2024-09-18T12:06:55.099413+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:55.099413+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:23.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:24.018 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11320/44 objects degraded (25727.273%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":843,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1744785408,"bytes_avail":288165507072,"bytes_total":289910292480,"degraded_objects":11320,"degraded_total":44,"degraded_ratio":257.27272727272725,"recovering_objects_per_sec":84,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1017,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:24.032 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:24.032 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:24.298 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:24.301 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:24.312 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-18T12:07:23.318652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31274,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1703892,"kb_used_data":160116,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281411628,"statfs":{"total":289910292480,"available":288165507072,"internally_reserved":0,"allocated":163958784,"data_stored":163051456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003879"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19621,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099497+0000","last_change":"2024-09-18T11:55:04.779429+0000","last_active":"2024-09-18T12:06:55.099497+0000","last_peered":"2024-09-18T12:06:55.099497+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:06:55.099497+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19763,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101021+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:06:55.101021+0000","last_peered":"2024-09-18T12:06:55.101021+0000","last_clean":"2024-09-18T12:06:55.101021+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:06:55.101021+0000","last_undegraded":"2024-09-18T12:06:55.101021+0000","last_fullsized":"2024-09-18T12:06:55.101021+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":25284,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:19.407803+0000","last_change":"2024-09-18T12:06:54.866165+0000","last_active":"2024-09-18T12:07:19.407803+0000","last_peered":"2024-09-18T12:07:19.407803+0000","last_clean":"2024-09-18T11:55:02.213764+0000","last_became_active":"2024-09-18T11:55:04.764665+0000","last_became_peered":"2024-09-18T11:55:04.764665+0000","last_unstale":"2024-09-18T12:07:19.407803+0000","last_undegraded":"2024-09-18T11:55:04.354089+0000","last_fullsized":"2024-09-18T11:55:04.248775+0000","mapping_epoch":36,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100566+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:06:55.100566+0000","last_peered":"2024-09-18T12:06:55.100566+0000","last_clean":"2024-09-18T12:06:55.100566+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:06:55.100566+0000","last_undegraded":"2024-09-18T12:06:55.100566+0000","last_fullsized":"2024-09-18T12:06:55.100566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26547,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100983+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:06:55.100983+0000","last_peered":"2024-09-18T12:06:55.100983+0000","last_clean":"2024-09-18T12:06:55.100983+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:06:55.100983+0000","last_undegraded":"2024-09-18T12:06:55.100983+0000","last_fullsized":"2024-09-18T12:06:55.100983+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26669,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:14.505218+0000","last_change":"2024-09-18T12:07:14.505218+0000","last_active":"2024-09-18T12:07:14.505218+0000","last_peered":"2024-09-18T12:07:14.505218+0000","last_clean":"2024-09-18T12:07:14.505218+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:14.505218+0000","last_undegraded":"2024-09-18T12:07:14.505218+0000","last_fullsized":"2024-09-18T12:07:14.505218+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":85,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.100970+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:06:55.100970+0000","last_peered":"2024-09-18T12:06:55.100970+0000","last_clean":"2024-09-18T12:06:55.100970+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:06:55.100970+0000","last_undegraded":"2024-09-18T12:06:55.100970+0000","last_fullsized":"2024-09-18T12:06:55.100970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101536+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:06:55.101536+0000","last_peered":"2024-09-18T12:06:55.101536+0000","last_clean":"2024-09-18T12:06:55.101536+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:06:55.101536+0000","last_undegraded":"2024-09-18T12:06:55.101536+0000","last_fullsized":"2024-09-18T12:06:55.101536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26682,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101128+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:06:55.101128+0000","last_peered":"2024-09-18T12:06:55.101128+0000","last_clean":"2024-09-18T12:06:55.101128+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:06:55.101128+0000","last_undegraded":"2024-09-18T12:06:55.101128+0000","last_fullsized":"2024-09-18T12:06:55.101128+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19300,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101113+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:06:55.101113+0000","last_peered":"2024-09-18T12:06:55.101113+0000","last_clean":"2024-09-18T12:06:55.101113+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:06:55.101113+0000","last_undegraded":"2024-09-18T12:06:55.101113+0000","last_fullsized":"2024-09-18T12:06:55.101113+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101085+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:06:55.101085+0000","last_peered":"2024-09-18T12:06:55.101085+0000","last_clean":"2024-09-18T12:06:55.101085+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:06:55.101085+0000","last_undegraded":"2024-09-18T12:06:55.101085+0000","last_fullsized":"2024-09-18T12:06:55.101085+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.101101+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:06:55.101101+0000","last_peered":"2024-09-18T12:06:55.101101+0000","last_clean":"2024-09-18T12:06:55.101101+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:06:55.101101+0000","last_undegraded":"2024-09-18T12:06:55.101101+0000","last_fullsized":"2024-09-18T12:06:55.101101+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19612,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:06:56.207646+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:06:56.207646+0000","last_peered":"2024-09-18T12:06:56.207646+0000","last_clean":"2024-09-18T12:06:56.207646+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:06:56.207646+0000","last_undegraded":"2024-09-18T12:06:56.207646+0000","last_fullsized":"2024-09-18T12:06:56.207646+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31273,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":162258944,"data_stored":162238070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051745,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":408788,"kb_used_data":102604,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93963052,"statfs":{"total":96636764160,"available":96218165248,"internally_reserved":0,"allocated":105066496,"data_stored":104772665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 12:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":1.345,"15min":1.6659999999999999},"min":{"1min":0.29199999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.69999999999999996,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":1.234,"15min":1.0920000000000001},"min":{"1min":0.35399999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.72699999999999998,"5min":12.712,"15min":12.712},"last":0.36099999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855508,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444084,"statfs":{"total":96636764160,"available":95686742016,"internally_reserved":0,"allocated":462848,"data_stored":150881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.38300000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.41099999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.69099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104259584,"data_stored":104256105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:27.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:27.654 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:27.666 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-18T12:07:27.319827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10663,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31931,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1693192,"kb_used_data":157732,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281422328,"statfs":{"total":289910292480,"available":288176463872,"internally_reserved":0,"allocated":161517568,"data_stored":160599421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002348"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19624,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.412285+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:24.412285+0000","last_peered":"2024-09-18T12:07:24.412285+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:24.412285+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3598}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:23.508425+0000","last_change":"2024-09-18T12:07:23.508425+0000","last_active":"2024-09-18T12:07:23.508425+0000","last_peered":"2024-09-18T12:07:23.508425+0000","last_clean":"2024-09-18T12:07:23.508425+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:23.508425+0000","last_undegraded":"2024-09-18T12:07:23.508425+0000","last_fullsized":"2024-09-18T12:07:23.508425+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10663,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31930,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159801344,"data_stored":159780470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051746,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406388,"kb_used_data":100204,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93965452,"statfs":{"total":96636764160,"available":96220622848,"internally_reserved":0,"allocated":102608896,"data_stored":102315065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.375},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33800000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.495},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.51800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855509,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919456,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452384,"statfs":{"total":96636764160,"available":95695241216,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.371},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.439}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.37},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101801984,"data_stored":101798505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:27.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:27.975 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:27.978 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:27.990 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-18T12:07:27.319827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10663,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31931,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1693192,"kb_used_data":157732,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281422328,"statfs":{"total":289910292480,"available":288176463872,"internally_reserved":0,"allocated":161517568,"data_stored":160599421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002348"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19624,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.412285+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:24.412285+0000","last_peered":"2024-09-18T12:07:24.412285+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:24.412285+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3598}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:23.508425+0000","last_change":"2024-09-18T12:07:23.508425+0000","last_active":"2024-09-18T12:07:23.508425+0000","last_peered":"2024-09-18T12:07:23.508425+0000","last_clean":"2024-09-18T12:07:23.508425+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:23.508425+0000","last_undegraded":"2024-09-18T12:07:23.508425+0000","last_fullsized":"2024-09-18T12:07:23.508425+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10663,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31930,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159801344,"data_stored":159780470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051746,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406388,"kb_used_data":100204,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93965452,"statfs":{"total":96636764160,"available":96220622848,"internally_reserved":0,"allocated":102608896,"data_stored":102315065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.375},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33800000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.495},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.51800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855509,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919456,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452384,"statfs":{"total":96636764160,"available":95695241216,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.371},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.439}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.37},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101801984,"data_stored":101798505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:27.992 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:27.992 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 19624, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.412285+0000', 'last_change': '2024-09-18T12:07:24.395137+0000', 'last_active': '2024-09-18T12:07:24.412285+0000', 'last_peered': '2024-09-18T12:07:24.412285+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:07:24.412285+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3598, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3598}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:27.992 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:27.992 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19518, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.097395+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:06:55.097395+0000', 'last_peered': '2024-09-18T12:06:55.097395+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:06:55.097395+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:27.992 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:27.992 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19624, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:06:55.099413+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:06:55.099413+0000', 'last_peered': '2024-09-18T12:06:55.099413+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:06:55.099413+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:27.992 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:28.379 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:28.379 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10663/44 objects degraded (24234.091%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":848,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1733828608,"bytes_avail":288176463872,"bytes_total":289910292480,"degraded_objects":10663,"degraded_total":44,"degraded_ratio":242.34090909090909,"recovering_objects_per_sec":150,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1202,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:28.392 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:28.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:28.696 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:28.698 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:28.710 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-18T12:07:27.319827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10663,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31931,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1693192,"kb_used_data":157732,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281422328,"statfs":{"total":289910292480,"available":288176463872,"internally_reserved":0,"allocated":161517568,"data_stored":160599421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002348"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":19624,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.412285+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:24.412285+0000","last_peered":"2024-09-18T12:07:24.412285+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:24.412285+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3598}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19604,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103713+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:06:55.103713+0000","last_peered":"2024-09-18T12:06:55.103713+0000","last_clean":"2024-09-18T12:06:55.103713+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:06:55.103713+0000","last_undegraded":"2024-09-18T12:06:55.103713+0000","last_fullsized":"2024-09-18T12:06:55.103713+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19303,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098530+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:06:55.098530+0000","last_peered":"2024-09-18T12:06:55.098530+0000","last_clean":"2024-09-18T12:06:55.098530+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:06:55.098530+0000","last_undegraded":"2024-09-18T12:06:55.098530+0000","last_fullsized":"2024-09-18T12:06:55.098530+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19652,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097101+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:06:55.097101+0000","last_peered":"2024-09-18T12:06:55.097101+0000","last_clean":"2024-09-18T12:06:55.097101+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:06:55.097101+0000","last_undegraded":"2024-09-18T12:06:55.097101+0000","last_fullsized":"2024-09-18T12:06:55.097101+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19518,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.097395+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:06:55.097395+0000","last_peered":"2024-09-18T12:06:55.097395+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:06:55.097395+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19841,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096569+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:06:55.096569+0000","last_peered":"2024-09-18T12:06:55.096569+0000","last_clean":"2024-09-18T12:06:55.096569+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:06:55.096569+0000","last_undegraded":"2024-09-18T12:06:55.096569+0000","last_fullsized":"2024-09-18T12:06:55.096569+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.097295+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:06:55.097295+0000","last_peered":"2024-09-18T12:06:55.097295+0000","last_clean":"2024-09-18T12:06:55.097295+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:06:55.097295+0000","last_undegraded":"2024-09-18T12:06:55.097295+0000","last_fullsized":"2024-09-18T12:06:55.097295+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26617,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.098454+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:06:55.098454+0000","last_peered":"2024-09-18T12:06:55.098454+0000","last_clean":"2024-09-18T12:06:55.098454+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:06:55.098454+0000","last_undegraded":"2024-09-18T12:06:55.098454+0000","last_fullsized":"2024-09-18T12:06:55.098454+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096869+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:06:55.096869+0000","last_peered":"2024-09-18T12:06:55.096869+0000","last_clean":"2024-09-18T12:06:55.096869+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:06:55.096869+0000","last_undegraded":"2024-09-18T12:06:55.096869+0000","last_fullsized":"2024-09-18T12:06:55.096869+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-18T12:07:23.508425+0000","last_change":"2024-09-18T12:07:23.508425+0000","last_active":"2024-09-18T12:07:23.508425+0000","last_peered":"2024-09-18T12:07:23.508425+0000","last_clean":"2024-09-18T12:07:23.508425+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:23.508425+0000","last_undegraded":"2024-09-18T12:07:23.508425+0000","last_fullsized":"2024-09-18T12:07:23.508425+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26925,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.096859+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:06:55.096859+0000","last_peered":"2024-09-18T12:06:55.096859+0000","last_clean":"2024-09-18T12:06:55.096859+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:06:55.096859+0000","last_undegraded":"2024-09-18T12:06:55.096859+0000","last_fullsized":"2024-09-18T12:06:55.096859+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19624,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:06:55.099413+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:06:55.099413+0000","last_peered":"2024-09-18T12:06:55.099413+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:06:55.099413+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-18T12:06:55.103584+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:06:55.103584+0000","last_peered":"2024-09-18T12:06:55.103584+0000","last_clean":"2024-09-18T12:06:55.103584+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:06:55.103584+0000","last_undegraded":"2024-09-18T12:06:55.103584+0000","last_fullsized":"2024-09-18T12:06:55.103584+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10663,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31930,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159801344,"data_stored":159780470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051746,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406388,"kb_used_data":100204,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93965452,"statfs":{"total":96636764160,"available":96220622848,"internally_reserved":0,"allocated":102608896,"data_stored":102315065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.375},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33800000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.495},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.51800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855509,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919456,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452384,"statfs":{"total":96636764160,"available":95695241216,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.371},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.439}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367348,"kb_used_data":57060,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004492,"statfs":{"total":96636764160,"available":96260599808,"internally_reserved":0,"allocated":58429440,"data_stored":58127910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.37},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101801984,"data_stored":101798505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:31.712 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:32.066 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:32.069 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:32.082 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-18T12:07:31.321187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32587,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1690592,"kb_used_data":155128,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281424928,"statfs":{"total":289910292480,"available":288179126272,"internally_reserved":0,"allocated":158851072,"data_stored":157911287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002534"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":20938,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:29.415289+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:29.415289+0000","last_peered":"2024-09-18T12:07:29.415289+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:29.415289+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2942}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26680,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.501572+0000","last_change":"2024-09-18T12:07:25.501553+0000","last_active":"2024-09-18T12:07:25.501572+0000","last_peered":"2024-09-18T12:07:25.501572+0000","last_clean":"2024-09-18T12:07:25.501572+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:25.501572+0000","last_undegraded":"2024-09-18T12:07:25.501572+0000","last_fullsized":"2024-09-18T12:07:25.501572+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32586,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":156979200,"data_stored":156958326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051747,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403772,"kb_used_data":97584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93968068,"statfs":{"total":96636764160,"available":96223301632,"internally_reserved":0,"allocated":99926016,"data_stored":99621366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.45000000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.496}]}]},{"osd":1,"up_from":35,"seq":150323855509,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919456,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452384,"statfs":{"total":96636764160,"available":95695241216,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.371},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.439}]}]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.27600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.434}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98979840,"data_stored":98976361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:32.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:32.406 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:32.418 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-18T12:07:31.321187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32587,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1690592,"kb_used_data":155128,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281424928,"statfs":{"total":289910292480,"available":288179126272,"internally_reserved":0,"allocated":158851072,"data_stored":157911287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002534"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":20938,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:29.415289+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:29.415289+0000","last_peered":"2024-09-18T12:07:29.415289+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:29.415289+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2942}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26680,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.501572+0000","last_change":"2024-09-18T12:07:25.501553+0000","last_active":"2024-09-18T12:07:25.501572+0000","last_peered":"2024-09-18T12:07:25.501572+0000","last_clean":"2024-09-18T12:07:25.501572+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:25.501572+0000","last_undegraded":"2024-09-18T12:07:25.501572+0000","last_fullsized":"2024-09-18T12:07:25.501572+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32586,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":156979200,"data_stored":156958326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051747,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403772,"kb_used_data":97584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93968068,"statfs":{"total":96636764160,"available":96223301632,"internally_reserved":0,"allocated":99926016,"data_stored":99621366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.45000000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.496}]}]},{"osd":1,"up_from":35,"seq":150323855509,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919456,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452384,"statfs":{"total":96636764160,"available":95695241216,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.371},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.439}]}]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.27600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.434}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98979840,"data_stored":98976361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:32.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:32.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 20938, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:29.415289+0000', 'last_change': '2024-09-18T12:07:24.395137+0000', 'last_active': '2024-09-18T12:07:29.415289+0000', 'last_peered': '2024-09-18T12:07:29.415289+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:07:29.415289+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2942, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 657, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2942}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:32.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:32.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19520, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.530967+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:07:24.530967+0000', 'last_peered': '2024-09-18T12:07:24.530967+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:07:24.530967+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:32.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:32.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19626, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.533203+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:24.533203+0000', 'last_peered': '2024-09-18T12:07:24.533203+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:24.533203+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:32.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:32.771 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:32.772 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10007/44 objects degraded (22743.182%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":852,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1731166208,"bytes_avail":288179126272,"bytes_total":289910292480,"degraded_objects":10007,"degraded_total":44,"degraded_ratio":227.43181818181819,"recovering_objects_per_sec":164,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1313,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:32.786 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:32.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:33.098 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:33.100 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:33.110 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-18T12:07:31.321187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32587,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1690592,"kb_used_data":155128,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281424928,"statfs":{"total":289910292480,"available":288179126272,"internally_reserved":0,"allocated":158851072,"data_stored":157911287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002534"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":20938,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:29.415289+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:29.415289+0000","last_peered":"2024-09-18T12:07:29.415289+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:29.415289+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2942}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26680,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.501572+0000","last_change":"2024-09-18T12:07:25.501553+0000","last_active":"2024-09-18T12:07:25.501572+0000","last_peered":"2024-09-18T12:07:25.501572+0000","last_clean":"2024-09-18T12:07:25.501572+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:25.501572+0000","last_undegraded":"2024-09-18T12:07:25.501572+0000","last_fullsized":"2024-09-18T12:07:25.501572+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10007,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32586,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":156979200,"data_stored":156958326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051747,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403772,"kb_used_data":97584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93968068,"statfs":{"total":96636764160,"available":96223301632,"internally_reserved":0,"allocated":99926016,"data_stored":99621366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.45000000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.496}]}]},{"osd":1,"up_from":35,"seq":150323855509,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919456,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452384,"statfs":{"total":96636764160,"available":95695241216,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.50600000000000001,"15min":0.45500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.55200000000000005,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.371},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.0049999999999999,"15min":1.21},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.63500000000000001,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.284,"15min":1.3200000000000001},"min":{"1min":0.27400000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":13.737,"15min":13.737},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.79500000000000004,"15min":0.76100000000000001},"min":{"1min":0.314,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.59099999999999997,"5min":8.4730000000000008,"15min":11.996},"last":0.439}]}]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.81799999999999995,"15min":0.98799999999999999},"min":{"1min":0.215,"5min":0.186,"15min":0.161},"max":{"1min":0.52400000000000002,"5min":11.521000000000001,"15min":11.521000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.61699999999999999,"15min":0.90500000000000003},"min":{"1min":0.249,"5min":0.189,"15min":0.189},"max":{"1min":0.59499999999999997,"5min":5.875,"15min":8.3460000000000001},"last":0.27600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.434}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98979840,"data_stored":98976361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:36.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:36.428 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:36.439 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-18T12:07:35.322171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33267,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1691744,"kb_used_data":152128,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281423776,"statfs":{"total":289910292480,"available":288177946624,"internally_reserved":0,"allocated":155779072,"data_stored":154839287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002900"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":22298,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:34.419976+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:34.419976+0000","last_peered":"2024-09-18T12:07:34.419976+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:34.419976+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26686,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:33.431361+0000","last_change":"2024-09-18T12:07:33.431361+0000","last_active":"2024-09-18T12:07:33.431361+0000","last_peered":"2024-09-18T12:07:33.431361+0000","last_clean":"2024-09-18T12:07:33.431361+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:33.431361+0000","last_undegraded":"2024-09-18T12:07:33.431361+0000","last_fullsized":"2024-09-18T12:07:33.431361+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33266,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":153907200,"data_stored":153886326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051748,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400768,"kb_used_data":94584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93971072,"statfs":{"total":96636764160,"available":96226377728,"internally_reserved":0,"allocated":96854016,"data_stored":96549366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.314},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.442}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.41199999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855510,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923612,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448228,"statfs":{"total":96636764160,"available":95690985472,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.30399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.377}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.222},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.33700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95907840,"data_stored":95904361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:36.440 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:36.746 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:36.748 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:36.766 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-18T12:07:35.322171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33267,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1691744,"kb_used_data":152128,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281423776,"statfs":{"total":289910292480,"available":288177946624,"internally_reserved":0,"allocated":155779072,"data_stored":154839287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002900"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":22298,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:34.419976+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:34.419976+0000","last_peered":"2024-09-18T12:07:34.419976+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:34.419976+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26686,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:33.431361+0000","last_change":"2024-09-18T12:07:33.431361+0000","last_active":"2024-09-18T12:07:33.431361+0000","last_peered":"2024-09-18T12:07:33.431361+0000","last_clean":"2024-09-18T12:07:33.431361+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:33.431361+0000","last_undegraded":"2024-09-18T12:07:33.431361+0000","last_fullsized":"2024-09-18T12:07:33.431361+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33266,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":153907200,"data_stored":153886326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051748,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400768,"kb_used_data":94584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93971072,"statfs":{"total":96636764160,"available":96226377728,"internally_reserved":0,"allocated":96854016,"data_stored":96549366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.314},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.442}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.41199999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855510,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923612,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448228,"statfs":{"total":96636764160,"available":95690985472,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.30399999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.377}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.222},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.33700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95907840,"data_stored":95904361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:36.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:36.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 22298, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:34.419976+0000', 'last_change': '2024-09-18T12:07:24.395137+0000', 'last_active': '2024-09-18T12:07:34.419976+0000', 'last_peered': '2024-09-18T12:07:34.419976+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:07:34.419976+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1337, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2262}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:36.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:36.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19520, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.530967+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:07:24.530967+0000', 'last_peered': '2024-09-18T12:07:24.530967+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:07:24.530967+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:36.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:36.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19626, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.533203+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:24.533203+0000', 'last_peered': '2024-09-18T12:07:24.533203+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:24.533203+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:36.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:37.135 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:37.136 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9327/44 objects degraded (21197.727%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":857,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1732345856,"bytes_avail":288177946624,"bytes_total":289910292480,"degraded_objects":9327,"degraded_total":44,"degraded_ratio":211.97727272727272,"recovering_objects_per_sec":133,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1336,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:37.155 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:37.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:37.456 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:37.471 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-18T12:07:37.322750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33267,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1695904,"kb_used_data":152128,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281419616,"statfs":{"total":289910292480,"available":288173686784,"internally_reserved":0,"allocated":155779072,"data_stored":154839287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.779761"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":22298,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:34.419976+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:34.419976+0000","last_peered":"2024-09-18T12:07:34.419976+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:34.419976+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26686,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:33.431361+0000","last_change":"2024-09-18T12:07:33.431361+0000","last_active":"2024-09-18T12:07:33.431361+0000","last_peered":"2024-09-18T12:07:33.431361+0000","last_clean":"2024-09-18T12:07:33.431361+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:33.431361+0000","last_undegraded":"2024-09-18T12:07:33.431361+0000","last_fullsized":"2024-09-18T12:07:33.431361+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33266,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":153907200,"data_stored":153886326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051748,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400768,"kb_used_data":94584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93971072,"statfs":{"total":96636764160,"available":96226377728,"internally_reserved":0,"allocated":96854016,"data_stored":96549366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.314},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.442}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.41199999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855511,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444068,"statfs":{"total":96636764160,"available":95686725632,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.432},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.38600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.222},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.33700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95907840,"data_stored":95904361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:40.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:40.848 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:40.850 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:40.862 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-09-18T12:07:39.323186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33267,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1692904,"kb_used_data":149128,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281422616,"statfs":{"total":289910292480,"available":288176758784,"internally_reserved":0,"allocated":152707072,"data_stored":151767287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003359"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":22298,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:34.419976+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:34.419976+0000","last_peered":"2024-09-18T12:07:34.419976+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:34.419976+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26689,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:35.352965+0000","last_change":"2024-09-18T12:07:35.352965+0000","last_active":"2024-09-18T12:07:35.352965+0000","last_peered":"2024-09-18T12:07:35.352965+0000","last_clean":"2024-09-18T12:07:35.352965+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:35.352965+0000","last_undegraded":"2024-09-18T12:07:35.352965+0000","last_fullsized":"2024-09-18T12:07:35.352965+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33266,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150835200,"data_stored":150814326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051749,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397768,"kb_used_data":91584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93974072,"statfs":{"total":96636764160,"available":96229449728,"internally_reserved":0,"allocated":93782016,"data_stored":93477366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33000000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.442},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.40799999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855511,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444068,"statfs":{"total":96636764160,"available":95686725632,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.432},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.38600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.222},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.33700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":92835840,"data_stored":92832361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:40.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:41.165 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:41.167 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:41.177 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-09-18T12:07:39.323186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33267,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1692904,"kb_used_data":149128,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281422616,"statfs":{"total":289910292480,"available":288176758784,"internally_reserved":0,"allocated":152707072,"data_stored":151767287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003359"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":22298,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:34.419976+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:34.419976+0000","last_peered":"2024-09-18T12:07:34.419976+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:34.419976+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26689,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:35.352965+0000","last_change":"2024-09-18T12:07:35.352965+0000","last_active":"2024-09-18T12:07:35.352965+0000","last_peered":"2024-09-18T12:07:35.352965+0000","last_clean":"2024-09-18T12:07:35.352965+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:35.352965+0000","last_undegraded":"2024-09-18T12:07:35.352965+0000","last_fullsized":"2024-09-18T12:07:35.352965+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33266,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150835200,"data_stored":150814326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051749,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397768,"kb_used_data":91584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93974072,"statfs":{"total":96636764160,"available":96229449728,"internally_reserved":0,"allocated":93782016,"data_stored":93477366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33000000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.442},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.40799999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855511,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444068,"statfs":{"total":96636764160,"available":95686725632,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.432},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.38600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.222},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.33700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":92835840,"data_stored":92832361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:41.179 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:41.179 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 22298, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:34.419976+0000', 'last_change': '2024-09-18T12:07:24.395137+0000', 'last_active': '2024-09-18T12:07:34.419976+0000', 'last_peered': '2024-09-18T12:07:34.419976+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:07:34.419976+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1337, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2262}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:41.179 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:41.179 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19520, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.530967+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:07:24.530967+0000', 'last_peered': '2024-09-18T12:07:24.530967+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:07:24.530967+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:41.179 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:41.179 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19626, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.533203+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:24.533203+0000', 'last_peered': '2024-09-18T12:07:24.533203+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:24.533203+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:41.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:41.610 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8843/44 objects degraded (20097.727%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":861,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1733533696,"bytes_avail":288176758784,"bytes_total":289910292480,"degraded_objects":8843,"degraded_total":44,"degraded_ratio":200.97727272727272,"recovering_objects_per_sec":151,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1820,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:41.623 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:41.623 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:41.912 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:41.915 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:41.927 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-18T12:07:41.323791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33751,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1692904,"kb_used_data":149128,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281422616,"statfs":{"total":289910292480,"available":288176758784,"internally_reserved":0,"allocated":152707072,"data_stored":151767287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1820,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003216"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":23266,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:39.426678+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:39.426678+0000","last_peered":"2024-09-18T12:07:39.426678+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:39.426678+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1778}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26689,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:35.352965+0000","last_change":"2024-09-18T12:07:35.352965+0000","last_active":"2024-09-18T12:07:35.352965+0000","last_peered":"2024-09-18T12:07:35.352965+0000","last_clean":"2024-09-18T12:07:35.352965+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:35.352965+0000","last_undegraded":"2024-09-18T12:07:35.352965+0000","last_fullsized":"2024-09-18T12:07:35.352965+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33750,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150835200,"data_stored":150814326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051749,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397768,"kb_used_data":91584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93974072,"statfs":{"total":96636764160,"available":96229449728,"internally_reserved":0,"allocated":93782016,"data_stored":93477366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33000000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.442},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.40799999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855511,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927772,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444068,"statfs":{"total":96636764160,"available":95686725632,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.432},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.38600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.33800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":92835840,"data_stored":92832361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:44.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:45.207 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:45.210 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:45.223 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-18T12:07:43.324569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33751,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1680424,"kb_used_data":149128,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281435096,"statfs":{"total":289910292480,"available":288189538304,"internally_reserved":0,"allocated":152707072,"data_stored":151767287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003382"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":23266,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:39.426678+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:39.426678+0000","last_peered":"2024-09-18T12:07:39.426678+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:39.426678+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1778,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1778}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26689,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:35.352965+0000","last_change":"2024-09-18T12:07:35.352965+0000","last_active":"2024-09-18T12:07:35.352965+0000","last_peered":"2024-09-18T12:07:35.352965+0000","last_clean":"2024-09-18T12:07:35.352965+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:35.352965+0000","last_undegraded":"2024-09-18T12:07:35.352965+0000","last_fullsized":"2024-09-18T12:07:35.352965+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33750,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150835200,"data_stored":150814326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051749,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397768,"kb_used_data":91584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93974072,"statfs":{"total":96636764160,"available":96229449728,"internally_reserved":0,"allocated":93782016,"data_stored":93477366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:06:37 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.89000000000000001,"15min":0.71299999999999997},"min":{"1min":0.28499999999999998,"5min":0.17000000000000001,"15min":0.153},"max":{"1min":0.61399999999999999,"5min":19.776,"15min":19.776},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":1.482,"15min":1.1539999999999999},"min":{"1min":0.29399999999999998,"5min":0.184,"15min":0.17499999999999999},"max":{"1min":0.59099999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.33000000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.442},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.40799999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855512,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915292,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456548,"statfs":{"total":96636764160,"available":95699505152,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.47399999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.495},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.63800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.33800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":92835840,"data_stored":92832361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:45.224 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:45.521 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:45.523 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:45.534 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-18T12:07:45.325115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34205,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1677424,"kb_used_data":146128,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281438096,"statfs":{"total":289910292480,"available":288192610304,"internally_reserved":0,"allocated":149635072,"data_stored":148695287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1618,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1618,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003396"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":24174,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:44.423384+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:44.423384+0000","last_peered":"2024-09-18T12:07:44.423384+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:44.423384+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1324}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26698,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:43.391192+0000","last_change":"2024-09-18T12:07:43.391192+0000","last_active":"2024-09-18T12:07:43.391192+0000","last_peered":"2024-09-18T12:07:43.391192+0000","last_clean":"2024-09-18T12:07:43.391192+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:43.391192+0000","last_undegraded":"2024-09-18T12:07:43.391192+0000","last_fullsized":"2024-09-18T12:07:43.391192+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34204,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147763200,"data_stored":147742326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051750,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394768,"kb_used_data":88584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93977072,"statfs":{"total":96636764160,"available":96232521728,"internally_reserved":0,"allocated":90710016,"data_stored":90405366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38700000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.54600000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855512,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915292,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456548,"statfs":{"total":96636764160,"available":95699505152,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.47399999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.495},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.63800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.35399999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.64300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89763840,"data_stored":89760361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:45.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:45.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 24174, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:44.423384+0000', 'last_change': '2024-09-18T12:07:24.395137+0000', 'last_active': '2024-09-18T12:07:44.423384+0000', 'last_peered': '2024-09-18T12:07:44.423384+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:07:44.423384+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1324, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2275, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1324}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:45.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:45.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19520, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.530967+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:07:24.530967+0000', 'last_peered': '2024-09-18T12:07:24.530967+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:07:24.530967+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:45.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:45.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19626, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.533203+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:24.533203+0000', 'last_peered': '2024-09-18T12:07:24.533203+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:24.533203+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:45.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:45.892 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8389/44 objects degraded (19065.909%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":865,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1717682176,"bytes_avail":288192610304,"bytes_total":289910292480,"degraded_objects":8389,"degraded_total":44,"degraded_ratio":190.65909090909091,"recovering_objects_per_sec":134,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1618,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:45.909 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:45.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:46.194 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:46.197 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:46.209 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-18T12:07:45.325115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34205,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1677424,"kb_used_data":146128,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281438096,"statfs":{"total":289910292480,"available":288192610304,"internally_reserved":0,"allocated":149635072,"data_stored":148695287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1618,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1618,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003396"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":24174,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:44.423384+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:44.423384+0000","last_peered":"2024-09-18T12:07:44.423384+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:44.423384+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1324}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26698,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:43.391192+0000","last_change":"2024-09-18T12:07:43.391192+0000","last_active":"2024-09-18T12:07:43.391192+0000","last_peered":"2024-09-18T12:07:43.391192+0000","last_clean":"2024-09-18T12:07:43.391192+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:43.391192+0000","last_undegraded":"2024-09-18T12:07:43.391192+0000","last_fullsized":"2024-09-18T12:07:43.391192+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34204,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147763200,"data_stored":147742326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051750,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394768,"kb_used_data":88584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93977072,"statfs":{"total":96636764160,"available":96232521728,"internally_reserved":0,"allocated":90710016,"data_stored":90405366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38700000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.54600000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855512,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915292,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456548,"statfs":{"total":96636764160,"available":95699505152,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.47399999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.495},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.63800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.35399999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.64300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89763840,"data_stored":89760361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:49.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:49.519 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:49.520 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:49.531 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-18T12:07:49.326257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34205,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1678584,"kb_used_data":143128,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281436936,"statfs":{"total":289910292480,"available":288191422464,"internally_reserved":0,"allocated":146563072,"data_stored":145623287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003507"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":24174,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:44.423384+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:44.423384+0000","last_peered":"2024-09-18T12:07:44.423384+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:44.423384+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1324}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26707,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:47.480899+0000","last_change":"2024-09-18T12:07:47.480899+0000","last_active":"2024-09-18T12:07:47.480899+0000","last_peered":"2024-09-18T12:07:47.480899+0000","last_clean":"2024-09-18T12:07:47.480899+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:47.480899+0000","last_undegraded":"2024-09-18T12:07:47.480899+0000","last_fullsized":"2024-09-18T12:07:47.480899+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34204,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144691200,"data_stored":144670326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051751,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391768,"kb_used_data":85584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93980072,"statfs":{"total":96636764160,"available":96235593728,"internally_reserved":0,"allocated":87638016,"data_stored":87333366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41299999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.39800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855513,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919452,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452388,"statfs":{"total":96636764160,"available":95695245312,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.51000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.35399999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.64300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86691840,"data_stored":86688361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:49.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:49.820 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:49.823 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:49.834 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-18T12:07:49.326257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34205,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1678584,"kb_used_data":143128,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281436936,"statfs":{"total":289910292480,"available":288191422464,"internally_reserved":0,"allocated":146563072,"data_stored":145623287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003507"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":24174,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:44.423384+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:44.423384+0000","last_peered":"2024-09-18T12:07:44.423384+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:44.423384+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1324}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26707,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:47.480899+0000","last_change":"2024-09-18T12:07:47.480899+0000","last_active":"2024-09-18T12:07:47.480899+0000","last_peered":"2024-09-18T12:07:47.480899+0000","last_clean":"2024-09-18T12:07:47.480899+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:47.480899+0000","last_undegraded":"2024-09-18T12:07:47.480899+0000","last_fullsized":"2024-09-18T12:07:47.480899+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34204,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144691200,"data_stored":144670326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051751,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391768,"kb_used_data":85584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93980072,"statfs":{"total":96636764160,"available":96235593728,"internally_reserved":0,"allocated":87638016,"data_stored":87333366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41299999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.39800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855513,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919452,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452388,"statfs":{"total":96636764160,"available":95695245312,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.51000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.35399999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.64300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86691840,"data_stored":86688361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:49.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:49.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 24174, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:44.423384+0000', 'last_change': '2024-09-18T12:07:24.395137+0000', 'last_active': '2024-09-18T12:07:44.423384+0000', 'last_peered': '2024-09-18T12:07:44.423384+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:07:44.423384+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1324, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2275, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1324}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:49.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:49.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19520, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.530967+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:07:24.530967+0000', 'last_peered': '2024-09-18T12:07:24.530967+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:07:24.530967+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:49.836 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:49.836 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19626, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.533203+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:24.533203+0000', 'last_peered': '2024-09-18T12:07:24.533203+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:24.533203+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:49.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:50.185 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:50.185 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8389/44 objects degraded (19065.909%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":870,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1718870016,"bytes_avail":288191422464,"bytes_total":289910292480,"degraded_objects":8389,"degraded_total":44,"degraded_ratio":190.65909090909091,"recovering_objects_per_sec":78,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":938,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:50.199 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:50.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:50.498 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:50.500 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:50.510 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-18T12:07:49.326257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34205,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1678584,"kb_used_data":143128,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281436936,"statfs":{"total":289910292480,"available":288191422464,"internally_reserved":0,"allocated":146563072,"data_stored":145623287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-938,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003507"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":24174,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:44.423384+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:44.423384+0000","last_peered":"2024-09-18T12:07:44.423384+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:44.423384+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1324}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26707,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:47.480899+0000","last_change":"2024-09-18T12:07:47.480899+0000","last_active":"2024-09-18T12:07:47.480899+0000","last_peered":"2024-09-18T12:07:47.480899+0000","last_clean":"2024-09-18T12:07:47.480899+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:47.480899+0000","last_undegraded":"2024-09-18T12:07:47.480899+0000","last_fullsized":"2024-09-18T12:07:47.480899+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34204,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144691200,"data_stored":144670326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051751,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391768,"kb_used_data":85584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93980072,"statfs":{"total":96636764160,"available":96235593728,"internally_reserved":0,"allocated":87638016,"data_stored":87333366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41299999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.39800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855513,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919452,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452388,"statfs":{"total":96636764160,"available":95695245312,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.51000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.35399999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.099,"15min":1.1339999999999999},"min":{"1min":0.25800000000000001,"5min":0.19500000000000001,"15min":0.16500000000000001},"max":{"1min":0.66800000000000004,"5min":11.071999999999999,"15min":13.773},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.95299999999999996,"15min":0.96699999999999997},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.17499999999999999},"max":{"1min":0.747,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.64300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86691840,"data_stored":86688361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:53.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:53.803 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:53.806 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:53.816 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-18T12:07:53.327285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34740,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1682744,"kb_used_data":143128,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281432776,"statfs":{"total":289910292480,"available":288187162624,"internally_reserved":0,"allocated":146563072,"data_stored":145623287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":989,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003493"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":25244,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:49.430710+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:49.430710+0000","last_peered":"2024-09-18T12:07:49.430710+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:49.430710+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":789,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2810,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-789}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26707,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:47.480899+0000","last_change":"2024-09-18T12:07:47.480899+0000","last_active":"2024-09-18T12:07:47.480899+0000","last_peered":"2024-09-18T12:07:47.480899+0000","last_clean":"2024-09-18T12:07:47.480899+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:47.480899+0000","last_undegraded":"2024-09-18T12:07:47.480899+0000","last_fullsized":"2024-09-18T12:07:47.480899+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34739,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144691200,"data_stored":144670326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051751,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391768,"kb_used_data":85584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93980072,"statfs":{"total":96636764160,"available":96235593728,"internally_reserved":0,"allocated":87638016,"data_stored":87333366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41299999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.39800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855514,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923612,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448228,"statfs":{"total":96636764160,"available":95690985472,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.375},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.40500000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.44},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.625}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86691840,"data_stored":86688361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:53.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:54.112 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:54.115 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:54.128 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-18T12:07:53.327285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34740,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1682744,"kb_used_data":143128,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281432776,"statfs":{"total":289910292480,"available":288187162624,"internally_reserved":0,"allocated":146563072,"data_stored":145623287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":989,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003493"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":25244,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:49.430710+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:49.430710+0000","last_peered":"2024-09-18T12:07:49.430710+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:49.430710+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":789,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2810,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-789}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26707,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:47.480899+0000","last_change":"2024-09-18T12:07:47.480899+0000","last_active":"2024-09-18T12:07:47.480899+0000","last_peered":"2024-09-18T12:07:47.480899+0000","last_clean":"2024-09-18T12:07:47.480899+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:47.480899+0000","last_undegraded":"2024-09-18T12:07:47.480899+0000","last_fullsized":"2024-09-18T12:07:47.480899+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34739,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144691200,"data_stored":144670326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051751,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391768,"kb_used_data":85584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93980072,"statfs":{"total":96636764160,"available":96235593728,"internally_reserved":0,"allocated":87638016,"data_stored":87333366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41299999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.39800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855514,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923612,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448228,"statfs":{"total":96636764160,"available":95690985472,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.375},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.40500000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.44},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.625}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86691840,"data_stored":86688361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:54.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:54.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 25244, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:49.430710+0000', 'last_change': '2024-09-18T12:07:24.395137+0000', 'last_active': '2024-09-18T12:07:49.430710+0000', 'last_peered': '2024-09-18T12:07:49.430710+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:07:49.430710+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 789, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2810, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -789}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:54.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:54.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19520, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.530967+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:07:24.530967+0000', 'last_peered': '2024-09-18T12:07:24.530967+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:07:24.530967+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:54.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:54.129 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19626, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.533203+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:24.533203+0000', 'last_peered': '2024-09-18T12:07:24.533203+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:24.533203+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:54.130 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:54.485 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7854/44 objects degraded (17850.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":874,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1723129856,"bytes_avail":288187162624,"bytes_total":289910292480,"degraded_objects":7854,"degraded_total":44,"degraded_ratio":178.5,"recovering_objects_per_sec":82,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":989,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:54.498 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:54.498 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:54.783 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:54.785 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:54.797 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-18T12:07:53.327285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34740,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1682744,"kb_used_data":143128,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281432776,"statfs":{"total":289910292480,"available":288187162624,"internally_reserved":0,"allocated":146563072,"data_stored":145623287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":989,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003493"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":25244,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:49.430710+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:49.430710+0000","last_peered":"2024-09-18T12:07:49.430710+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:49.430710+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":789,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2810,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-789}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19765,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537779+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:24.537779+0000","last_peered":"2024-09-18T12:07:24.537779+0000","last_clean":"2024-09-18T12:07:24.537779+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:24.537779+0000","last_undegraded":"2024-09-18T12:07:24.537779+0000","last_fullsized":"2024-09-18T12:07:24.537779+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19475,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.661526+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.661526+0000","last_peered":"2024-09-18T12:07:25.661526+0000","last_clean":"2024-09-18T12:07:25.661526+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.661526+0000","last_undegraded":"2024-09-18T12:07:25.661526+0000","last_fullsized":"2024-09-18T12:07:25.661526+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537465+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:24.537465+0000","last_peered":"2024-09-18T12:07:24.537465+0000","last_clean":"2024-09-18T12:07:24.537465+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:24.537465+0000","last_undegraded":"2024-09-18T12:07:24.537465+0000","last_fullsized":"2024-09-18T12:07:24.537465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26549,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537900+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:24.537900+0000","last_peered":"2024-09-18T12:07:24.537900+0000","last_clean":"2024-09-18T12:07:24.537900+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:24.537900+0000","last_undegraded":"2024-09-18T12:07:24.537900+0000","last_fullsized":"2024-09-18T12:07:24.537900+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26707,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:47.480899+0000","last_change":"2024-09-18T12:07:47.480899+0000","last_active":"2024-09-18T12:07:47.480899+0000","last_peered":"2024-09-18T12:07:47.480899+0000","last_clean":"2024-09-18T12:07:47.480899+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:47.480899+0000","last_undegraded":"2024-09-18T12:07:47.480899+0000","last_fullsized":"2024-09-18T12:07:47.480899+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":87,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537803+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:24.537803+0000","last_peered":"2024-09-18T12:07:24.537803+0000","last_clean":"2024-09-18T12:07:24.537803+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:24.537803+0000","last_undegraded":"2024-09-18T12:07:24.537803+0000","last_fullsized":"2024-09-18T12:07:24.537803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537994+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:24.537994+0000","last_peered":"2024-09-18T12:07:24.537994+0000","last_clean":"2024-09-18T12:07:24.537994+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:24.537994+0000","last_undegraded":"2024-09-18T12:07:24.537994+0000","last_fullsized":"2024-09-18T12:07:24.537994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26684,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537858+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:24.537858+0000","last_peered":"2024-09-18T12:07:24.537858+0000","last_clean":"2024-09-18T12:07:24.537858+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:24.537858+0000","last_undegraded":"2024-09-18T12:07:24.537858+0000","last_fullsized":"2024-09-18T12:07:24.537858+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19302,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537915+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:24.537915+0000","last_peered":"2024-09-18T12:07:24.537915+0000","last_clean":"2024-09-18T12:07:24.537915+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:24.537915+0000","last_undegraded":"2024-09-18T12:07:24.537915+0000","last_fullsized":"2024-09-18T12:07:24.537915+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":71,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537768+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:24.537768+0000","last_peered":"2024-09-18T12:07:24.537768+0000","last_clean":"2024-09-18T12:07:24.537768+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:24.537768+0000","last_undegraded":"2024-09-18T12:07:24.537768+0000","last_fullsized":"2024-09-18T12:07:24.537768+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.538107+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:24.538107+0000","last_peered":"2024-09-18T12:07:24.538107+0000","last_clean":"2024-09-18T12:07:24.538107+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:24.538107+0000","last_undegraded":"2024-09-18T12:07:24.538107+0000","last_fullsized":"2024-09-18T12:07:24.538107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19614,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.537895+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:24.537895+0000","last_peered":"2024-09-18T12:07:24.537895+0000","last_clean":"2024-09-18T12:07:24.537895+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:24.537895+0000","last_undegraded":"2024-09-18T12:07:24.537895+0000","last_fullsized":"2024-09-18T12:07:24.537895+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34739,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":144691200,"data_stored":144670326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051751,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391768,"kb_used_data":85584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93980072,"statfs":{"total":96636764160,"available":96235593728,"internally_reserved":0,"allocated":87638016,"data_stored":87333366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41299999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.39800000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855514,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923612,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448228,"statfs":{"total":96636764160,"available":95690985472,"internally_reserved":0,"allocated":479232,"data_stored":156446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.375},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.40500000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.44},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.625}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86691840,"data_stored":86688361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:57.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:58.099 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:58.101 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:58.112 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-18T12:07:57.328474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35370,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1688068,"kb_used_data":140128,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281427452,"statfs":{"total":289910292480,"available":288181710848,"internally_reserved":0,"allocated":143491072,"data_stored":142552821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002216"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26504,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:54.423784+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:54.423784+0000","last_peered":"2024-09-18T12:07:54.423784+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:54.423784+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3440,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-159}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19766,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545487+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:25.545487+0000","last_peered":"2024-09-18T12:07:25.545487+0000","last_clean":"2024-09-18T12:07:25.545487+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:25.545487+0000","last_undegraded":"2024-09-18T12:07:25.545487+0000","last_fullsized":"2024-09-18T12:07:25.545487+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19476,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.665191+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.665191+0000","last_peered":"2024-09-18T12:07:25.665191+0000","last_clean":"2024-09-18T12:07:25.665191+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.665191+0000","last_undegraded":"2024-09-18T12:07:25.665191+0000","last_fullsized":"2024-09-18T12:07:25.665191+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.544928+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:25.544928+0000","last_peered":"2024-09-18T12:07:25.544928+0000","last_clean":"2024-09-18T12:07:25.544928+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:25.544928+0000","last_undegraded":"2024-09-18T12:07:25.544928+0000","last_fullsized":"2024-09-18T12:07:25.544928+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26550,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545285+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:25.545285+0000","last_peered":"2024-09-18T12:07:25.545285+0000","last_clean":"2024-09-18T12:07:25.545285+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:25.545285+0000","last_undegraded":"2024-09-18T12:07:25.545285+0000","last_fullsized":"2024-09-18T12:07:25.545285+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26713,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:51.424549+0000","last_change":"2024-09-18T12:07:51.424549+0000","last_active":"2024-09-18T12:07:51.424549+0000","last_peered":"2024-09-18T12:07:51.424549+0000","last_clean":"2024-09-18T12:07:51.424549+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:51.424549+0000","last_undegraded":"2024-09-18T12:07:51.424549+0000","last_fullsized":"2024-09-18T12:07:51.424549+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":88,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545228+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:25.545228+0000","last_peered":"2024-09-18T12:07:25.545228+0000","last_clean":"2024-09-18T12:07:25.545228+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:25.545228+0000","last_undegraded":"2024-09-18T12:07:25.545228+0000","last_fullsized":"2024-09-18T12:07:25.545228+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545419+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:25.545419+0000","last_peered":"2024-09-18T12:07:25.545419+0000","last_clean":"2024-09-18T12:07:25.545419+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:25.545419+0000","last_undegraded":"2024-09-18T12:07:25.545419+0000","last_fullsized":"2024-09-18T12:07:25.545419+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26685,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545209+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:25.545209+0000","last_peered":"2024-09-18T12:07:25.545209+0000","last_clean":"2024-09-18T12:07:25.545209+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:25.545209+0000","last_undegraded":"2024-09-18T12:07:25.545209+0000","last_fullsized":"2024-09-18T12:07:25.545209+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19303,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545538+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:25.545538+0000","last_peered":"2024-09-18T12:07:25.545538+0000","last_clean":"2024-09-18T12:07:25.545538+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:25.545538+0000","last_undegraded":"2024-09-18T12:07:25.545538+0000","last_fullsized":"2024-09-18T12:07:25.545538+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545077+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:25.545077+0000","last_peered":"2024-09-18T12:07:25.545077+0000","last_clean":"2024-09-18T12:07:25.545077+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:25.545077+0000","last_undegraded":"2024-09-18T12:07:25.545077+0000","last_fullsized":"2024-09-18T12:07:25.545077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545633+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:25.545633+0000","last_peered":"2024-09-18T12:07:25.545633+0000","last_clean":"2024-09-18T12:07:25.545633+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:25.545633+0000","last_undegraded":"2024-09-18T12:07:25.545633+0000","last_fullsized":"2024-09-18T12:07:25.545633+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19615,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545366+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:25.545366+0000","last_peered":"2024-09-18T12:07:25.545366+0000","last_clean":"2024-09-18T12:07:25.545366+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:25.545366+0000","last_undegraded":"2024-09-18T12:07:25.545366+0000","last_fullsized":"2024-09-18T12:07:25.545366+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35369,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":141611008,"data_stored":141597092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051752,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388768,"kb_used_data":82584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93983072,"statfs":{"total":96636764160,"available":96238665728,"internally_reserved":0,"allocated":84566016,"data_stored":84261366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.32300000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.434},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.34999999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855515,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931936,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931451,"kb_avail":93439904,"statfs":{"total":96636764160,"available":95682461696,"internally_reserved":0,"allocated":479232,"data_stored":157980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":953806342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.375},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.32100000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83619840,"data_stored":83616361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:58.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:58.393 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:58.396 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:58.406 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-18T12:07:57.328474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35370,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1688068,"kb_used_data":140128,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281427452,"statfs":{"total":289910292480,"available":288181710848,"internally_reserved":0,"allocated":143491072,"data_stored":142552821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002216"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26504,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:54.423784+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:54.423784+0000","last_peered":"2024-09-18T12:07:54.423784+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:54.423784+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3440,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-159}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19766,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545487+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:25.545487+0000","last_peered":"2024-09-18T12:07:25.545487+0000","last_clean":"2024-09-18T12:07:25.545487+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:25.545487+0000","last_undegraded":"2024-09-18T12:07:25.545487+0000","last_fullsized":"2024-09-18T12:07:25.545487+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19476,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.665191+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.665191+0000","last_peered":"2024-09-18T12:07:25.665191+0000","last_clean":"2024-09-18T12:07:25.665191+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.665191+0000","last_undegraded":"2024-09-18T12:07:25.665191+0000","last_fullsized":"2024-09-18T12:07:25.665191+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.544928+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:25.544928+0000","last_peered":"2024-09-18T12:07:25.544928+0000","last_clean":"2024-09-18T12:07:25.544928+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:25.544928+0000","last_undegraded":"2024-09-18T12:07:25.544928+0000","last_fullsized":"2024-09-18T12:07:25.544928+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26550,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545285+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:25.545285+0000","last_peered":"2024-09-18T12:07:25.545285+0000","last_clean":"2024-09-18T12:07:25.545285+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:25.545285+0000","last_undegraded":"2024-09-18T12:07:25.545285+0000","last_fullsized":"2024-09-18T12:07:25.545285+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26713,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:51.424549+0000","last_change":"2024-09-18T12:07:51.424549+0000","last_active":"2024-09-18T12:07:51.424549+0000","last_peered":"2024-09-18T12:07:51.424549+0000","last_clean":"2024-09-18T12:07:51.424549+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:51.424549+0000","last_undegraded":"2024-09-18T12:07:51.424549+0000","last_fullsized":"2024-09-18T12:07:51.424549+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":88,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545228+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:25.545228+0000","last_peered":"2024-09-18T12:07:25.545228+0000","last_clean":"2024-09-18T12:07:25.545228+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:25.545228+0000","last_undegraded":"2024-09-18T12:07:25.545228+0000","last_fullsized":"2024-09-18T12:07:25.545228+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545419+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:25.545419+0000","last_peered":"2024-09-18T12:07:25.545419+0000","last_clean":"2024-09-18T12:07:25.545419+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:25.545419+0000","last_undegraded":"2024-09-18T12:07:25.545419+0000","last_fullsized":"2024-09-18T12:07:25.545419+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26685,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545209+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:25.545209+0000","last_peered":"2024-09-18T12:07:25.545209+0000","last_clean":"2024-09-18T12:07:25.545209+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:25.545209+0000","last_undegraded":"2024-09-18T12:07:25.545209+0000","last_fullsized":"2024-09-18T12:07:25.545209+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19303,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545538+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:25.545538+0000","last_peered":"2024-09-18T12:07:25.545538+0000","last_clean":"2024-09-18T12:07:25.545538+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:25.545538+0000","last_undegraded":"2024-09-18T12:07:25.545538+0000","last_fullsized":"2024-09-18T12:07:25.545538+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545077+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:25.545077+0000","last_peered":"2024-09-18T12:07:25.545077+0000","last_clean":"2024-09-18T12:07:25.545077+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:25.545077+0000","last_undegraded":"2024-09-18T12:07:25.545077+0000","last_fullsized":"2024-09-18T12:07:25.545077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545633+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:25.545633+0000","last_peered":"2024-09-18T12:07:25.545633+0000","last_clean":"2024-09-18T12:07:25.545633+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:25.545633+0000","last_undegraded":"2024-09-18T12:07:25.545633+0000","last_fullsized":"2024-09-18T12:07:25.545633+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19615,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545366+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:25.545366+0000","last_peered":"2024-09-18T12:07:25.545366+0000","last_clean":"2024-09-18T12:07:25.545366+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:25.545366+0000","last_undegraded":"2024-09-18T12:07:25.545366+0000","last_fullsized":"2024-09-18T12:07:25.545366+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35369,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":141611008,"data_stored":141597092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051752,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388768,"kb_used_data":82584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93983072,"statfs":{"total":96636764160,"available":96238665728,"internally_reserved":0,"allocated":84566016,"data_stored":84261366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.32300000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.434},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.34999999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855515,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931936,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931451,"kb_avail":93439904,"statfs":{"total":96636764160,"available":95682461696,"internally_reserved":0,"allocated":479232,"data_stored":157980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":953806342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.375},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.32100000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83619840,"data_stored":83616361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:07:58.407 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-18T12:07:58.407 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "43'19536", 'reported_seq': 26504, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:54.423784+0000', 'last_change': '2024-09-18T12:07:24.395137+0000', 'last_active': '2024-09-18T12:07:54.423784+0000', 'last_peered': '2024-09-18T12:07:54.423784+0000', 'last_clean': '2024-09-18T11:55:02.216000+0000', 'last_became_active': '2024-09-18T11:55:04.767559+0000', 'last_became_peered': '2024-09-18T11:55:04.767559+0000', 'last_unstale': '2024-09-18T12:07:54.423784+0000', 'last_undegraded': '2024-09-18T11:55:04.337874+0000', 'last_fullsized': '2024-09-18T11:55:04.252733+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 159, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19536, 'num_write_kb': 39072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3440, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -159}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:58.407 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:07:58.408 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 19520, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.530967+0000', 'last_change': '2024-09-18T11:55:04.785878+0000', 'last_active': '2024-09-18T12:07:24.530967+0000', 'last_peered': '2024-09-18T12:07:24.530967+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:07:24.530967+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3529, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:58.408 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:07:58.408 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19626, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:24.533203+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:24.533203+0000', 'last_peered': '2024-09-18T12:07:24.533203+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:24.533203+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:07:58.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:07:58.771 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:58.772 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7224/44 objects degraded (16418.182%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":878,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1728581632,"bytes_avail":288181710848,"bytes_total":289910292480,"degraded_objects":7224,"degraded_total":44,"degraded_ratio":164.18181818181819,"recovering_objects_per_sec":145,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1165,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:07:58.785 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:07:58.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:07:59.103 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:07:59.105 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:07:59.115 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-18T12:07:57.328474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35370,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1688068,"kb_used_data":140128,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281427452,"statfs":{"total":289910292480,"available":288181710848,"internally_reserved":0,"allocated":143491072,"data_stored":142552821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1165,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002216"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26504,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:54.423784+0000","last_change":"2024-09-18T12:07:24.395137+0000","last_active":"2024-09-18T12:07:54.423784+0000","last_peered":"2024-09-18T12:07:54.423784+0000","last_clean":"2024-09-18T11:55:02.216000+0000","last_became_active":"2024-09-18T11:55:04.767559+0000","last_became_peered":"2024-09-18T11:55:04.767559+0000","last_unstale":"2024-09-18T12:07:54.423784+0000","last_undegraded":"2024-09-18T11:55:04.337874+0000","last_fullsized":"2024-09-18T11:55:04.252733+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:43:01.616639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3440,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-159}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19606,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542278+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:24.542278+0000","last_peered":"2024-09-18T12:07:24.542278+0000","last_clean":"2024-09-18T12:07:24.542278+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:24.542278+0000","last_undegraded":"2024-09-18T12:07:24.542278+0000","last_fullsized":"2024-09-18T12:07:24.542278+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19766,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545487+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:25.545487+0000","last_peered":"2024-09-18T12:07:25.545487+0000","last_clean":"2024-09-18T12:07:25.545487+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:25.545487+0000","last_undegraded":"2024-09-18T12:07:25.545487+0000","last_fullsized":"2024-09-18T12:07:25.545487+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19305,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542009+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:24.542009+0000","last_peered":"2024-09-18T12:07:24.542009+0000","last_clean":"2024-09-18T12:07:24.542009+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:24.542009+0000","last_undegraded":"2024-09-18T12:07:24.542009+0000","last_fullsized":"2024-09-18T12:07:24.542009+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19654,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.532362+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:24.532362+0000","last_peered":"2024-09-18T12:07:24.532362+0000","last_clean":"2024-09-18T12:07:24.532362+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:24.532362+0000","last_undegraded":"2024-09-18T12:07:24.532362+0000","last_fullsized":"2024-09-18T12:07:24.532362+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19520,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.530967+0000","last_change":"2024-09-18T11:55:04.785878+0000","last_active":"2024-09-18T12:07:24.530967+0000","last_peered":"2024-09-18T12:07:24.530967+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:24.530967+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19476,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.665191+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.665191+0000","last_peered":"2024-09-18T12:07:25.665191+0000","last_clean":"2024-09-18T12:07:25.665191+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.665191+0000","last_undegraded":"2024-09-18T12:07:25.665191+0000","last_fullsized":"2024-09-18T12:07:25.665191+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19843,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530384+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:24.530384+0000","last_peered":"2024-09-18T12:07:24.530384+0000","last_clean":"2024-09-18T12:07:24.530384+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:24.530384+0000","last_undegraded":"2024-09-18T12:07:24.530384+0000","last_fullsized":"2024-09-18T12:07:24.530384+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.544928+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:25.544928+0000","last_peered":"2024-09-18T12:07:25.544928+0000","last_clean":"2024-09-18T12:07:25.544928+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:25.544928+0000","last_undegraded":"2024-09-18T12:07:25.544928+0000","last_fullsized":"2024-09-18T12:07:25.544928+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26550,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545285+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:25.545285+0000","last_peered":"2024-09-18T12:07:25.545285+0000","last_clean":"2024-09-18T12:07:25.545285+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:25.545285+0000","last_undegraded":"2024-09-18T12:07:25.545285+0000","last_fullsized":"2024-09-18T12:07:25.545285+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542366+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:24.542366+0000","last_peered":"2024-09-18T12:07:24.542366+0000","last_clean":"2024-09-18T12:07:24.542366+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:24.542366+0000","last_undegraded":"2024-09-18T12:07:24.542366+0000","last_fullsized":"2024-09-18T12:07:24.542366+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26619,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.531989+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:24.531989+0000","last_peered":"2024-09-18T12:07:24.531989+0000","last_clean":"2024-09-18T12:07:24.531989+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:24.531989+0000","last_undegraded":"2024-09-18T12:07:24.531989+0000","last_fullsized":"2024-09-18T12:07:24.531989+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.530434+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:24.530434+0000","last_peered":"2024-09-18T12:07:24.530434+0000","last_clean":"2024-09-18T12:07:24.530434+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:24.530434+0000","last_undegraded":"2024-09-18T12:07:24.530434+0000","last_fullsized":"2024-09-18T12:07:24.530434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26713,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:51.424549+0000","last_change":"2024-09-18T12:07:51.424549+0000","last_active":"2024-09-18T12:07:51.424549+0000","last_peered":"2024-09-18T12:07:51.424549+0000","last_clean":"2024-09-18T12:07:51.424549+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:51.424549+0000","last_undegraded":"2024-09-18T12:07:51.424549+0000","last_fullsized":"2024-09-18T12:07:51.424549+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":88,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545228+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:25.545228+0000","last_peered":"2024-09-18T12:07:25.545228+0000","last_clean":"2024-09-18T12:07:25.545228+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:25.545228+0000","last_undegraded":"2024-09-18T12:07:25.545228+0000","last_fullsized":"2024-09-18T12:07:25.545228+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26927,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.541918+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:24.541918+0000","last_peered":"2024-09-18T12:07:24.541918+0000","last_clean":"2024-09-18T12:07:24.541918+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:24.541918+0000","last_undegraded":"2024-09-18T12:07:24.541918+0000","last_fullsized":"2024-09-18T12:07:24.541918+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19626,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:24.533203+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:24.533203+0000","last_peered":"2024-09-18T12:07:24.533203+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:24.533203+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545419+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:25.545419+0000","last_peered":"2024-09-18T12:07:25.545419+0000","last_clean":"2024-09-18T12:07:25.545419+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:25.545419+0000","last_undegraded":"2024-09-18T12:07:25.545419+0000","last_fullsized":"2024-09-18T12:07:25.545419+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26685,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545209+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:25.545209+0000","last_peered":"2024-09-18T12:07:25.545209+0000","last_clean":"2024-09-18T12:07:25.545209+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:25.545209+0000","last_undegraded":"2024-09-18T12:07:25.545209+0000","last_fullsized":"2024-09-18T12:07:25.545209+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-18T12:07:24.542198+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:24.542198+0000","last_peered":"2024-09-18T12:07:24.542198+0000","last_clean":"2024-09-18T12:07:24.542198+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:24.542198+0000","last_undegraded":"2024-09-18T12:07:24.542198+0000","last_fullsized":"2024-09-18T12:07:24.542198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19303,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545538+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:25.545538+0000","last_peered":"2024-09-18T12:07:25.545538+0000","last_clean":"2024-09-18T12:07:25.545538+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:25.545538+0000","last_undegraded":"2024-09-18T12:07:25.545538+0000","last_fullsized":"2024-09-18T12:07:25.545538+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545077+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:25.545077+0000","last_peered":"2024-09-18T12:07:25.545077+0000","last_clean":"2024-09-18T12:07:25.545077+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:25.545077+0000","last_undegraded":"2024-09-18T12:07:25.545077+0000","last_fullsized":"2024-09-18T12:07:25.545077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545633+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:25.545633+0000","last_peered":"2024-09-18T12:07:25.545633+0000","last_clean":"2024-09-18T12:07:25.545633+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:25.545633+0000","last_undegraded":"2024-09-18T12:07:25.545633+0000","last_fullsized":"2024-09-18T12:07:25.545633+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19615,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545366+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:25.545366+0000","last_peered":"2024-09-18T12:07:25.545366+0000","last_clean":"2024-09-18T12:07:25.545366+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:25.545366+0000","last_undegraded":"2024-09-18T12:07:25.545366+0000","last_fullsized":"2024-09-18T12:07:25.545366+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35369,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":141611008,"data_stored":141597092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051752,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388768,"kb_used_data":82584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93983072,"statfs":{"total":96636764160,"available":96238665728,"internally_reserved":0,"allocated":84566016,"data_stored":84261366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.32300000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.434},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.34999999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855515,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931936,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931451,"kb_avail":93439904,"statfs":{"total":96636764160,"available":95682461696,"internally_reserved":0,"allocated":479232,"data_stored":157980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":953806342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.375},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.32100000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367364,"kb_used_data":57076,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004476,"statfs":{"total":96636764160,"available":96260583424,"internally_reserved":0,"allocated":58445824,"data_stored":58133475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83619840,"data_stored":83616361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:02.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:02.444 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:02.446 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:02.457 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-09-18T12:08:01.329504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35895,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1684864,"kb_used_data":136924,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281430656,"statfs":{"total":289910292480,"available":288184991744,"internally_reserved":0,"allocated":140210176,"data_stored":139250187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002219"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19766,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545487+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:25.545487+0000","last_peered":"2024-09-18T12:07:25.545487+0000","last_clean":"2024-09-18T12:07:25.545487+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:25.545487+0000","last_undegraded":"2024-09-18T12:07:25.545487+0000","last_fullsized":"2024-09-18T12:07:25.545487+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":20255,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:59.428172+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:07:59.428172+0000","last_peered":"2024-09-18T12:07:59.428172+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:59.428172+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3161}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19476,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.665191+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.665191+0000","last_peered":"2024-09-18T12:07:25.665191+0000","last_clean":"2024-09-18T12:07:25.665191+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.665191+0000","last_undegraded":"2024-09-18T12:07:25.665191+0000","last_fullsized":"2024-09-18T12:07:25.665191+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.544928+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:25.544928+0000","last_peered":"2024-09-18T12:07:25.544928+0000","last_clean":"2024-09-18T12:07:25.544928+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:25.544928+0000","last_undegraded":"2024-09-18T12:07:25.544928+0000","last_fullsized":"2024-09-18T12:07:25.544928+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26550,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545285+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:25.545285+0000","last_peered":"2024-09-18T12:07:25.545285+0000","last_clean":"2024-09-18T12:07:25.545285+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:25.545285+0000","last_undegraded":"2024-09-18T12:07:25.545285+0000","last_fullsized":"2024-09-18T12:07:25.545285+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26721,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:58.404174+0000","last_change":"2024-09-18T12:07:58.404174+0000","last_active":"2024-09-18T12:07:58.404174+0000","last_peered":"2024-09-18T12:07:58.404174+0000","last_clean":"2024-09-18T12:07:58.404174+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:58.404174+0000","last_undegraded":"2024-09-18T12:07:58.404174+0000","last_fullsized":"2024-09-18T12:07:58.404174+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":88,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545228+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:25.545228+0000","last_peered":"2024-09-18T12:07:25.545228+0000","last_clean":"2024-09-18T12:07:25.545228+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:25.545228+0000","last_undegraded":"2024-09-18T12:07:25.545228+0000","last_fullsized":"2024-09-18T12:07:25.545228+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545419+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:25.545419+0000","last_peered":"2024-09-18T12:07:25.545419+0000","last_clean":"2024-09-18T12:07:25.545419+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:25.545419+0000","last_undegraded":"2024-09-18T12:07:25.545419+0000","last_fullsized":"2024-09-18T12:07:25.545419+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26685,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545209+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:25.545209+0000","last_peered":"2024-09-18T12:07:25.545209+0000","last_clean":"2024-09-18T12:07:25.545209+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:25.545209+0000","last_undegraded":"2024-09-18T12:07:25.545209+0000","last_fullsized":"2024-09-18T12:07:25.545209+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19303,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545538+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:25.545538+0000","last_peered":"2024-09-18T12:07:25.545538+0000","last_clean":"2024-09-18T12:07:25.545538+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:25.545538+0000","last_undegraded":"2024-09-18T12:07:25.545538+0000","last_fullsized":"2024-09-18T12:07:25.545538+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545077+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:25.545077+0000","last_peered":"2024-09-18T12:07:25.545077+0000","last_clean":"2024-09-18T12:07:25.545077+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:25.545077+0000","last_undegraded":"2024-09-18T12:07:25.545077+0000","last_fullsized":"2024-09-18T12:07:25.545077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545633+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:25.545633+0000","last_peered":"2024-09-18T12:07:25.545633+0000","last_clean":"2024-09-18T12:07:25.545633+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:25.545633+0000","last_undegraded":"2024-09-18T12:07:25.545633+0000","last_fullsized":"2024-09-18T12:07:25.545633+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19615,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545366+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:25.545366+0000","last_peered":"2024-09-18T12:07:25.545366+0000","last_clean":"2024-09-18T12:07:25.545366+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:25.545366+0000","last_undegraded":"2024-09-18T12:07:25.545366+0000","last_fullsized":"2024-09-18T12:07:25.545366+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35894,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":138174464,"data_stored":138160548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051753,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385552,"kb_used_data":79364,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93986288,"statfs":{"total":96636764160,"available":96241958912,"internally_reserved":0,"allocated":81268736,"data_stored":80953217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.496}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.438},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.54800000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855515,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931936,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931451,"kb_avail":93439904,"statfs":{"total":96636764160,"available":95682461696,"internally_reserved":0,"allocated":479232,"data_stored":157980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":953806342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.375},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.32100000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":80183296,"data_stored":80179817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:02.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:02.763 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:02.765 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:02.775 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-09-18T12:08:01.329504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35895,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1684864,"kb_used_data":136924,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281430656,"statfs":{"total":289910292480,"available":288184991744,"internally_reserved":0,"allocated":140210176,"data_stored":139250187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002219"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19766,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545487+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:25.545487+0000","last_peered":"2024-09-18T12:07:25.545487+0000","last_clean":"2024-09-18T12:07:25.545487+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:25.545487+0000","last_undegraded":"2024-09-18T12:07:25.545487+0000","last_fullsized":"2024-09-18T12:07:25.545487+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":20255,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:59.428172+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:07:59.428172+0000","last_peered":"2024-09-18T12:07:59.428172+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:59.428172+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3161}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19476,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.665191+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:25.665191+0000","last_peered":"2024-09-18T12:07:25.665191+0000","last_clean":"2024-09-18T12:07:25.665191+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:25.665191+0000","last_undegraded":"2024-09-18T12:07:25.665191+0000","last_fullsized":"2024-09-18T12:07:25.665191+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.544928+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:25.544928+0000","last_peered":"2024-09-18T12:07:25.544928+0000","last_clean":"2024-09-18T12:07:25.544928+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:25.544928+0000","last_undegraded":"2024-09-18T12:07:25.544928+0000","last_fullsized":"2024-09-18T12:07:25.544928+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26550,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545285+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:25.545285+0000","last_peered":"2024-09-18T12:07:25.545285+0000","last_clean":"2024-09-18T12:07:25.545285+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:25.545285+0000","last_undegraded":"2024-09-18T12:07:25.545285+0000","last_fullsized":"2024-09-18T12:07:25.545285+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26721,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:58.404174+0000","last_change":"2024-09-18T12:07:58.404174+0000","last_active":"2024-09-18T12:07:58.404174+0000","last_peered":"2024-09-18T12:07:58.404174+0000","last_clean":"2024-09-18T12:07:58.404174+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:58.404174+0000","last_undegraded":"2024-09-18T12:07:58.404174+0000","last_fullsized":"2024-09-18T12:07:58.404174+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":88,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545228+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:25.545228+0000","last_peered":"2024-09-18T12:07:25.545228+0000","last_clean":"2024-09-18T12:07:25.545228+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:25.545228+0000","last_undegraded":"2024-09-18T12:07:25.545228+0000","last_fullsized":"2024-09-18T12:07:25.545228+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545419+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:25.545419+0000","last_peered":"2024-09-18T12:07:25.545419+0000","last_clean":"2024-09-18T12:07:25.545419+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:25.545419+0000","last_undegraded":"2024-09-18T12:07:25.545419+0000","last_fullsized":"2024-09-18T12:07:25.545419+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26685,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545209+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:25.545209+0000","last_peered":"2024-09-18T12:07:25.545209+0000","last_clean":"2024-09-18T12:07:25.545209+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:25.545209+0000","last_undegraded":"2024-09-18T12:07:25.545209+0000","last_fullsized":"2024-09-18T12:07:25.545209+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19303,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545538+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:25.545538+0000","last_peered":"2024-09-18T12:07:25.545538+0000","last_clean":"2024-09-18T12:07:25.545538+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:25.545538+0000","last_undegraded":"2024-09-18T12:07:25.545538+0000","last_fullsized":"2024-09-18T12:07:25.545538+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545077+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:25.545077+0000","last_peered":"2024-09-18T12:07:25.545077+0000","last_clean":"2024-09-18T12:07:25.545077+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:25.545077+0000","last_undegraded":"2024-09-18T12:07:25.545077+0000","last_fullsized":"2024-09-18T12:07:25.545077+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545633+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:25.545633+0000","last_peered":"2024-09-18T12:07:25.545633+0000","last_clean":"2024-09-18T12:07:25.545633+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:25.545633+0000","last_undegraded":"2024-09-18T12:07:25.545633+0000","last_fullsized":"2024-09-18T12:07:25.545633+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19615,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-18T12:07:25.545366+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:25.545366+0000","last_peered":"2024-09-18T12:07:25.545366+0000","last_clean":"2024-09-18T12:07:25.545366+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:25.545366+0000","last_undegraded":"2024-09-18T12:07:25.545366+0000","last_fullsized":"2024-09-18T12:07:25.545366+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35894,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":138174464,"data_stored":138160548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051753,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385552,"kb_used_data":79364,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93986288,"statfs":{"total":96636764160,"available":96241958912,"internally_reserved":0,"allocated":81268736,"data_stored":80953217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.496}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.438},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.54800000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855515,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931936,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931451,"kb_avail":93439904,"statfs":{"total":96636764160,"available":95682461696,"internally_reserved":0,"allocated":479232,"data_stored":157980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":953806342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.375},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.32100000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":80183296,"data_stored":80179817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:02.777 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:08:02.777 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 20255, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:59.428172+0000', 'last_change': '2024-09-18T12:07:55.558532+0000', 'last_active': '2024-09-18T12:07:59.428172+0000', 'last_peered': '2024-09-18T12:07:59.428172+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:07:59.428172+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3163, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 368, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3161}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:02.777 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:02.777 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19628, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:55.656223+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:55.656223+0000', 'last_peered': '2024-09-18T12:07:55.656223+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:55.656223+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:02.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:03.118 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:03.118 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6699/44 objects degraded (15225.000%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":883,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1725300736,"bytes_avail":288184991744,"bytes_total":289910292480,"degraded_objects":6699,"degraded_total":44,"degraded_ratio":152.25,"recovering_objects_per_sec":144,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1155,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:03.133 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:03.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:03.431 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:03.444 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-18T12:08:03.330127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35895,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1668228,"kb_used_data":136932,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281447292,"statfs":{"total":289910292480,"available":288202027008,"internally_reserved":0,"allocated":140218368,"data_stored":139252934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002227"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":20255,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:59.428172+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:07:59.428172+0000","last_peered":"2024-09-18T12:07:59.428172+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:07:59.428172+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3161}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26721,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:58.404174+0000","last_change":"2024-09-18T12:07:58.404174+0000","last_active":"2024-09-18T12:07:58.404174+0000","last_peered":"2024-09-18T12:07:58.404174+0000","last_clean":"2024-09-18T12:07:58.404174+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:07:58.404174+0000","last_undegraded":"2024-09-18T12:07:58.404174+0000","last_fullsized":"2024-09-18T12:07:58.404174+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35894,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":138174464,"data_stored":138160548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051753,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385552,"kb_used_data":79364,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93986288,"statfs":{"total":96636764160,"available":96241958912,"internally_reserved":0,"allocated":81268736,"data_stored":80953217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.496}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.438},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.54800000000000004}]}]},{"osd":1,"up_from":35,"seq":150323855516,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915300,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456540,"statfs":{"total":96636764160,"available":95699496960,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.44700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.499}]}]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":80183296,"data_stored":80179817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:06.446 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:06.740 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:06.743 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:06.753 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-18T12:08:05.330706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36261,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1664624,"kb_used_data":133332,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281450896,"statfs":{"total":289910292480,"available":288205717504,"internally_reserved":0,"allocated":136531968,"data_stored":135566534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-891,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":891,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.678033"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":20987,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:04.434563+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:04.434563+0000","last_peered":"2024-09-18T12:08:04.434563+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:04.434563+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":734,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2795}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26724,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:00.388122+0000","last_change":"2024-09-18T12:08:00.388122+0000","last_active":"2024-09-18T12:08:00.388122+0000","last_peered":"2024-09-18T12:08:00.388122+0000","last_clean":"2024-09-18T12:08:00.388122+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:00.388122+0000","last_undegraded":"2024-09-18T12:08:00.388122+0000","last_fullsized":"2024-09-18T12:08:00.388122+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36260,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134733824,"data_stored":134719908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051754,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381948,"kb_used_data":75764,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93989892,"statfs":{"total":96636764160,"available":96245649408,"internally_reserved":0,"allocated":77582336,"data_stored":77266817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.39900000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.63500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855516,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915300,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456540,"statfs":{"total":96636764160,"available":95699496960,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.44700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.499}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":76742656,"data_stored":76739177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:06.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:07.046 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:07.049 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:07.062 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-18T12:08:05.330706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36261,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1664624,"kb_used_data":133332,"kb_used_omap":13,"kb_used_meta":1531250,"kb_avail":281450896,"statfs":{"total":289910292480,"available":288205717504,"internally_reserved":0,"allocated":136531968,"data_stored":135566534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1568000530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-891,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":891,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.678033"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":20987,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:04.434563+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:04.434563+0000","last_peered":"2024-09-18T12:08:04.434563+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:04.434563+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":734,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2795}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26724,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:00.388122+0000","last_change":"2024-09-18T12:08:00.388122+0000","last_active":"2024-09-18T12:08:00.388122+0000","last_peered":"2024-09-18T12:08:00.388122+0000","last_clean":"2024-09-18T12:08:00.388122+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:00.388122+0000","last_undegraded":"2024-09-18T12:08:00.388122+0000","last_fullsized":"2024-09-18T12:08:00.388122+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36260,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134733824,"data_stored":134719908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051754,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381948,"kb_used_data":75764,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93989892,"statfs":{"total":96636764160,"available":96245649408,"internally_reserved":0,"allocated":77582336,"data_stored":77266817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.39900000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.63500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855516,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915300,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456540,"statfs":{"total":96636764160,"available":95699496960,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.44700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.499}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":76742656,"data_stored":76739177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:07.063 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:08:07.063 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 20987, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:04.434563+0000', 'last_change': '2024-09-18T12:07:55.558532+0000', 'last_active': '2024-09-18T12:08:04.434563+0000', 'last_peered': '2024-09-18T12:08:04.434563+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:08:04.434563+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2797, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 734, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2795}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:07.063 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:07.064 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19628, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:55.656223+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:55.656223+0000', 'last_peered': '2024-09-18T12:07:55.656223+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:55.656223+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:07.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:07.416 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:07.416 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6333/44 objects degraded (14393.182%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":887,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1704574976,"bytes_avail":288205717504,"bytes_total":289910292480,"degraded_objects":6333,"degraded_total":44,"degraded_ratio":143.93181818181819,"recovering_objects_per_sec":92,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":891,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:07.431 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:07.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:07.805 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:07.807 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:07.817 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-18T12:08:07.331231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36261,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1668784,"kb_used_data":133332,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281446736,"statfs":{"total":289910292480,"available":288201457664,"internally_reserved":0,"allocated":136531968,"data_stored":135566534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-891,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":891,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.674435"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":20987,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:04.434563+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:04.434563+0000","last_peered":"2024-09-18T12:08:04.434563+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:04.434563+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":734,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2795}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26724,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:00.388122+0000","last_change":"2024-09-18T12:08:00.388122+0000","last_active":"2024-09-18T12:08:00.388122+0000","last_peered":"2024-09-18T12:08:00.388122+0000","last_clean":"2024-09-18T12:08:00.388122+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:00.388122+0000","last_undegraded":"2024-09-18T12:08:00.388122+0000","last_fullsized":"2024-09-18T12:08:00.388122+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36260,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":134733824,"data_stored":134719908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051754,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381948,"kb_used_data":75764,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93989892,"statfs":{"total":96636764160,"available":96245649408,"internally_reserved":0,"allocated":77582336,"data_stored":77266817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.39900000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.63500000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855517,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919460,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452380,"statfs":{"total":96636764160,"available":95695237120,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.54600000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":76742656,"data_stored":76739177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:10.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:11.117 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:11.119 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:11.129 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-09-18T12:08:09.331686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36261,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1665304,"kb_used_data":129852,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281450216,"statfs":{"total":289910292480,"available":288205021184,"internally_reserved":0,"allocated":132968448,"data_stored":132006493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-891,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":891,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003212"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":20987,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:04.434563+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:04.434563+0000","last_peered":"2024-09-18T12:08:04.434563+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:04.434563+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":734,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2795}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26724,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:00.388122+0000","last_change":"2024-09-18T12:08:00.388122+0000","last_active":"2024-09-18T12:08:00.388122+0000","last_peered":"2024-09-18T12:08:00.388122+0000","last_clean":"2024-09-18T12:08:00.388122+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:00.388122+0000","last_undegraded":"2024-09-18T12:08:00.388122+0000","last_fullsized":"2024-09-18T12:08:00.388122+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36260,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131170304,"data_stored":131159867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051755,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378468,"kb_used_data":72284,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93993372,"statfs":{"total":96636764160,"available":96249212928,"internally_reserved":0,"allocated":74018816,"data_stored":73706776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.39600000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.36299999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855517,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919460,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452380,"statfs":{"total":96636764160,"available":95695237120,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.54600000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73179136,"data_stored":73179136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:11.130 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:11.497 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:11.500 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:11.511 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-18T12:08:11.332219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36638,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1665304,"kb_used_data":129852,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281450216,"statfs":{"total":289910292480,"available":288205021184,"internally_reserved":0,"allocated":132968448,"data_stored":132006493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1268,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003209"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":21741,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:09.433102+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:09.433102+0000","last_peered":"2024-09-18T12:08:09.433102+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:09.433102+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26724,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:00.388122+0000","last_change":"2024-09-18T12:08:00.388122+0000","last_active":"2024-09-18T12:08:00.388122+0000","last_peered":"2024-09-18T12:08:00.388122+0000","last_clean":"2024-09-18T12:08:00.388122+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:00.388122+0000","last_undegraded":"2024-09-18T12:08:00.388122+0000","last_fullsized":"2024-09-18T12:08:00.388122+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36637,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131170304,"data_stored":131159867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051755,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378468,"kb_used_data":72284,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93993372,"statfs":{"total":96636764160,"available":96249212928,"internally_reserved":0,"allocated":74018816,"data_stored":73706776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.39600000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.36299999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855517,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919460,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452380,"statfs":{"total":96636764160,"available":95695237120,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.54600000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73179136,"data_stored":73179136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:11.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:08:11.512 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 21741, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:09.433102+0000', 'last_change': '2024-09-18T12:07:55.558532+0000', 'last_active': '2024-09-18T12:08:09.433102+0000', 'last_peered': '2024-09-18T12:08:09.433102+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:08:09.433102+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2420, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2418}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:11.513 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:11.513 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19628, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:55.656223+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:55.656223+0000', 'last_peered': '2024-09-18T12:07:55.656223+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:55.656223+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:11.513 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:11.893 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:11.894 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5956/44 objects degraded (13536.364%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":891,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1705271296,"bytes_avail":288205021184,"bytes_total":289910292480,"degraded_objects":5956,"degraded_total":44,"degraded_ratio":135.36363636363637,"recovering_objects_per_sec":105,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1268,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:11.907 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:11.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:12.209 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:12.212 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:12.225 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-18T12:08:11.332219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36638,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1665304,"kb_used_data":129852,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281450216,"statfs":{"total":289910292480,"available":288205021184,"internally_reserved":0,"allocated":132968448,"data_stored":132006493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1268,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003209"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":21741,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:09.433102+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:09.433102+0000","last_peered":"2024-09-18T12:08:09.433102+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:09.433102+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26724,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:00.388122+0000","last_change":"2024-09-18T12:08:00.388122+0000","last_active":"2024-09-18T12:08:00.388122+0000","last_peered":"2024-09-18T12:08:00.388122+0000","last_clean":"2024-09-18T12:08:00.388122+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:00.388122+0000","last_undegraded":"2024-09-18T12:08:00.388122+0000","last_fullsized":"2024-09-18T12:08:00.388122+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36637,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131170304,"data_stored":131159867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051755,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":378468,"kb_used_data":72284,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93993372,"statfs":{"total":96636764160,"available":96249212928,"internally_reserved":0,"allocated":74018816,"data_stored":73706776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.39600000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.36299999999999999}]}]},{"osd":1,"up_from":35,"seq":150323855517,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919460,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452380,"statfs":{"total":96636764160,"available":95695237120,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.54600000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73179136,"data_stored":73179136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:15.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:15.540 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:15.542 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:15.554 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-09-18T12:08:15.333374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37046,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1665864,"kb_used_data":126252,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281449656,"statfs":{"total":289910292480,"available":288204447744,"internally_reserved":0,"allocated":129282048,"data_stored":128320093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003247"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":22557,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:14.439596+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:14.439596+0000","last_peered":"2024-09-18T12:08:14.439596+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:14.439596+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1519,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26724,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:00.388122+0000","last_change":"2024-09-18T12:08:00.388122+0000","last_active":"2024-09-18T12:08:00.388122+0000","last_peered":"2024-09-18T12:08:00.388122+0000","last_clean":"2024-09-18T12:08:00.388122+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:00.388122+0000","last_undegraded":"2024-09-18T12:08:00.388122+0000","last_fullsized":"2024-09-18T12:08:00.388122+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37045,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":127483904,"data_stored":127473467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051756,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374868,"kb_used_data":68684,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93996972,"statfs":{"total":96636764160,"available":96252899328,"internally_reserved":0,"allocated":70332416,"data_stored":70020376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.44}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.499},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.56000000000000005}]}]},{"osd":1,"up_from":35,"seq":150323855518,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923620,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448220,"statfs":{"total":96636764160,"available":95690977280,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.44},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.499}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.66500000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.312}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69492736,"data_stored":69492736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:15.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:15.841 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:15.844 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:15.855 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-09-18T12:08:15.333374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37046,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1665864,"kb_used_data":126252,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281449656,"statfs":{"total":289910292480,"available":288204447744,"internally_reserved":0,"allocated":129282048,"data_stored":128320093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003247"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":22557,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:14.439596+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:14.439596+0000","last_peered":"2024-09-18T12:08:14.439596+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:14.439596+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1519,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26724,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:00.388122+0000","last_change":"2024-09-18T12:08:00.388122+0000","last_active":"2024-09-18T12:08:00.388122+0000","last_peered":"2024-09-18T12:08:00.388122+0000","last_clean":"2024-09-18T12:08:00.388122+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:00.388122+0000","last_undegraded":"2024-09-18T12:08:00.388122+0000","last_fullsized":"2024-09-18T12:08:00.388122+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37045,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":127483904,"data_stored":127473467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051756,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374868,"kb_used_data":68684,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93996972,"statfs":{"total":96636764160,"available":96252899328,"internally_reserved":0,"allocated":70332416,"data_stored":70020376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.44}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.499},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.56000000000000005}]}]},{"osd":1,"up_from":35,"seq":150323855518,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923620,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448220,"statfs":{"total":96636764160,"available":95690977280,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.44},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.499}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.66500000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.312}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69492736,"data_stored":69492736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:15.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:08:15.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 22557, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:14.439596+0000', 'last_change': '2024-09-18T12:07:55.558532+0000', 'last_active': '2024-09-18T12:08:14.439596+0000', 'last_peered': '2024-09-18T12:08:14.439596+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:08:14.439596+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2012, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1519, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2010}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:15.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:15.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19628, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:55.656223+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:55.656223+0000', 'last_peered': '2024-09-18T12:07:55.656223+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:55.656223+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:15.856 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:16.212 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:16.212 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5548/44 objects degraded (12609.091%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":896,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1705844736,"bytes_avail":288204447744,"bytes_total":289910292480,"degraded_objects":5548,"degraded_total":44,"degraded_ratio":126.09090909090909,"recovering_objects_per_sec":95,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1151,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:16.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:16.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:16.530 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:16.532 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:16.542 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-09-18T12:08:15.333374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37046,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1665864,"kb_used_data":126252,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281449656,"statfs":{"total":289910292480,"available":288204447744,"internally_reserved":0,"allocated":129282048,"data_stored":128320093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003247"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":22557,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:14.439596+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:14.439596+0000","last_peered":"2024-09-18T12:08:14.439596+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:14.439596+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1519,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26724,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:00.388122+0000","last_change":"2024-09-18T12:08:00.388122+0000","last_active":"2024-09-18T12:08:00.388122+0000","last_peered":"2024-09-18T12:08:00.388122+0000","last_clean":"2024-09-18T12:08:00.388122+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:00.388122+0000","last_undegraded":"2024-09-18T12:08:00.388122+0000","last_fullsized":"2024-09-18T12:08:00.388122+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37045,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":127483904,"data_stored":127473467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051756,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":374868,"kb_used_data":68684,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93996972,"statfs":{"total":96636764160,"available":96252899328,"internally_reserved":0,"allocated":70332416,"data_stored":70020376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.44}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.499},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.56000000000000005}]}]},{"osd":1,"up_from":35,"seq":150323855518,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923620,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448220,"statfs":{"total":96636764160,"available":95690977280,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.44},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.499}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.66500000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.312}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69492736,"data_stored":69492736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:19.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:19.870 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:19.873 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:19.883 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-18T12:08:19.334449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37046,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1666424,"kb_used_data":122652,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281449096,"statfs":{"total":289910292480,"available":288203874304,"internally_reserved":0,"allocated":125595648,"data_stored":124633693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":785,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003218"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":22557,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:14.439596+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:14.439596+0000","last_peered":"2024-09-18T12:08:14.439596+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:14.439596+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1519,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26733,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:18.367992+0000","last_change":"2024-09-18T12:08:18.367992+0000","last_active":"2024-09-18T12:08:18.367992+0000","last_peered":"2024-09-18T12:08:18.367992+0000","last_clean":"2024-09-18T12:08:18.367992+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:18.367992+0000","last_undegraded":"2024-09-18T12:08:18.367992+0000","last_fullsized":"2024-09-18T12:08:18.367992+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37045,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":123797504,"data_stored":123787067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051757,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371268,"kb_used_data":65084,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94000572,"statfs":{"total":96636764160,"available":96256585728,"internally_reserved":0,"allocated":66646016,"data_stored":66333976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41099999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.371},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.504}]}]},{"osd":1,"up_from":35,"seq":150323855519,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927780,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444060,"statfs":{"total":96636764160,"available":95686717440,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.45100000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.312}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65806336,"data_stored":65806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:19.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:20.191 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:20.194 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:20.205 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-18T12:08:19.334449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37046,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1666424,"kb_used_data":122652,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281449096,"statfs":{"total":289910292480,"available":288203874304,"internally_reserved":0,"allocated":125595648,"data_stored":124633693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":785,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003218"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":22557,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:14.439596+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:14.439596+0000","last_peered":"2024-09-18T12:08:14.439596+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:14.439596+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1519,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26733,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:18.367992+0000","last_change":"2024-09-18T12:08:18.367992+0000","last_active":"2024-09-18T12:08:18.367992+0000","last_peered":"2024-09-18T12:08:18.367992+0000","last_clean":"2024-09-18T12:08:18.367992+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:18.367992+0000","last_undegraded":"2024-09-18T12:08:18.367992+0000","last_fullsized":"2024-09-18T12:08:18.367992+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37045,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":123797504,"data_stored":123787067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051757,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371268,"kb_used_data":65084,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94000572,"statfs":{"total":96636764160,"available":96256585728,"internally_reserved":0,"allocated":66646016,"data_stored":66333976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41099999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.371},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.504}]}]},{"osd":1,"up_from":35,"seq":150323855519,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927780,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444060,"statfs":{"total":96636764160,"available":95686717440,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.45100000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.312}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65806336,"data_stored":65806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:20.206 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:08:20.206 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 22557, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:14.439596+0000', 'last_change': '2024-09-18T12:07:55.558532+0000', 'last_active': '2024-09-18T12:08:14.439596+0000', 'last_peered': '2024-09-18T12:08:14.439596+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:08:14.439596+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2012, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1519, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2010}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:20.206 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:20.207 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19628, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:55.656223+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:55.656223+0000', 'last_peered': '2024-09-18T12:07:55.656223+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:55.656223+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:20.207 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:20.543 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:20.544 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5548/44 objects degraded (12609.091%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":900,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1706418176,"bytes_avail":288203874304,"bytes_total":289910292480,"degraded_objects":5548,"degraded_total":44,"degraded_ratio":126.09090909090909,"recovering_objects_per_sec":65,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":785,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:20.556 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:20.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:20.853 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:20.855 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:20.868 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-18T12:08:19.334449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37046,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1666424,"kb_used_data":122652,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281449096,"statfs":{"total":289910292480,"available":288203874304,"internally_reserved":0,"allocated":125595648,"data_stored":124633693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":785,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003218"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":22557,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:14.439596+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:14.439596+0000","last_peered":"2024-09-18T12:08:14.439596+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:14.439596+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1519,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26733,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:18.367992+0000","last_change":"2024-09-18T12:08:18.367992+0000","last_active":"2024-09-18T12:08:18.367992+0000","last_peered":"2024-09-18T12:08:18.367992+0000","last_clean":"2024-09-18T12:08:18.367992+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:18.367992+0000","last_undegraded":"2024-09-18T12:08:18.367992+0000","last_fullsized":"2024-09-18T12:08:18.367992+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37045,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":123797504,"data_stored":123787067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051757,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371268,"kb_used_data":65084,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94000572,"statfs":{"total":96636764160,"available":96256585728,"internally_reserved":0,"allocated":66646016,"data_stored":66333976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41099999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.371},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.504}]}]},{"osd":1,"up_from":35,"seq":150323855519,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927780,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444060,"statfs":{"total":96636764160,"available":95686717440,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.45100000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.312}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65806336,"data_stored":65806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:23.870 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:24.178 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:24.191 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-18T12:08:23.335564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37491,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1670584,"kb_used_data":122652,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281444936,"statfs":{"total":289910292480,"available":288199614464,"internally_reserved":0,"allocated":125595648,"data_stored":124633693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003344"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":23447,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:19.436645+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:19.436645+0000","last_peered":"2024-09-18T12:08:19.436645+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:19.436645+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1565}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26733,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:18.367992+0000","last_change":"2024-09-18T12:08:18.367992+0000","last_active":"2024-09-18T12:08:18.367992+0000","last_peered":"2024-09-18T12:08:18.367992+0000","last_clean":"2024-09-18T12:08:18.367992+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:18.367992+0000","last_undegraded":"2024-09-18T12:08:18.367992+0000","last_fullsized":"2024-09-18T12:08:18.367992+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37490,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":123797504,"data_stored":123787067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051757,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371268,"kb_used_data":65084,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94000572,"statfs":{"total":96636764160,"available":96256585728,"internally_reserved":0,"allocated":66646016,"data_stored":66333976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41099999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.371},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.504}]}]},{"osd":1,"up_from":35,"seq":150323855520,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931940,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":931451,"kb_avail":93439900,"statfs":{"total":96636764160,"available":95682457600,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":953806342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.43099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.48899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65806336,"data_stored":65806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:24.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:24.508 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:24.511 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:24.521 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-18T12:08:23.335564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37491,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1670584,"kb_used_data":122652,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281444936,"statfs":{"total":289910292480,"available":288199614464,"internally_reserved":0,"allocated":125595648,"data_stored":124633693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003344"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":23447,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:19.436645+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:19.436645+0000","last_peered":"2024-09-18T12:08:19.436645+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:19.436645+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1565}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26733,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:18.367992+0000","last_change":"2024-09-18T12:08:18.367992+0000","last_active":"2024-09-18T12:08:18.367992+0000","last_peered":"2024-09-18T12:08:18.367992+0000","last_clean":"2024-09-18T12:08:18.367992+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:18.367992+0000","last_undegraded":"2024-09-18T12:08:18.367992+0000","last_fullsized":"2024-09-18T12:08:18.367992+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37490,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":123797504,"data_stored":123787067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051757,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371268,"kb_used_data":65084,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94000572,"statfs":{"total":96636764160,"available":96256585728,"internally_reserved":0,"allocated":66646016,"data_stored":66333976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41099999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.371},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.504}]}]},{"osd":1,"up_from":35,"seq":150323855520,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931940,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":931451,"kb_avail":93439900,"statfs":{"total":96636764160,"available":95682457600,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":953806342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.43099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.48899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65806336,"data_stored":65806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:24.522 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:08:24.522 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 23447, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:19.436645+0000', 'last_change': '2024-09-18T12:07:55.558532+0000', 'last_active': '2024-09-18T12:08:19.436645+0000', 'last_peered': '2024-09-18T12:08:19.436645+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:08:19.436645+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1567, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1964, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1565}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:24.523 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:24.523 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19628, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:55.656223+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:55.656223+0000', 'last_peered': '2024-09-18T12:07:55.656223+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:55.656223+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:24.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:24.923 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:24.923 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5103/44 objects degraded (11597.727%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":904,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1710678016,"bytes_avail":288199614464,"bytes_total":289910292480,"degraded_objects":5103,"degraded_total":44,"degraded_ratio":115.97727272727273,"recovering_objects_per_sec":71,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":853,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:24.939 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:24.939 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:25.229 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:25.232 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:25.243 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-18T12:08:23.335564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37491,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1670584,"kb_used_data":122652,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281444936,"statfs":{"total":289910292480,"available":288199614464,"internally_reserved":0,"allocated":125595648,"data_stored":124633693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003344"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":23447,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:19.436645+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:19.436645+0000","last_peered":"2024-09-18T12:08:19.436645+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:19.436645+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1565}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26733,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:18.367992+0000","last_change":"2024-09-18T12:08:18.367992+0000","last_active":"2024-09-18T12:08:18.367992+0000","last_peered":"2024-09-18T12:08:18.367992+0000","last_clean":"2024-09-18T12:08:18.367992+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:18.367992+0000","last_undegraded":"2024-09-18T12:08:18.367992+0000","last_fullsized":"2024-09-18T12:08:18.367992+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37490,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":123797504,"data_stored":123787067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051757,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371268,"kb_used_data":65084,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94000572,"statfs":{"total":96636764160,"available":96256585728,"internally_reserved":0,"allocated":66646016,"data_stored":66333976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":313519622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.41099999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.1439999999999999,"15min":1.5569999999999999},"min":{"1min":0.29499999999999998,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.65400000000000003,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.371},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.88600000000000001,"15min":1.0609999999999999},"min":{"1min":0.26700000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":8.0449999999999999,"15min":12.712},"last":0.504}]}]},{"osd":1,"up_from":35,"seq":150323855520,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931940,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":931451,"kb_avail":93439900,"statfs":{"total":96636764160,"available":95682457600,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":953806342},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.43099999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.48899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.45600000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65806336,"data_stored":65806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:28.245 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:28.575 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:28.577 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:28.588 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-18T12:08:27.336774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37986,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1658664,"kb_used_data":119052,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281456856,"statfs":{"total":289910292480,"available":288211820544,"internally_reserved":0,"allocated":121909248,"data_stored":120947293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003400"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":24437,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:24.438283+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:24.438283+0000","last_peered":"2024-09-18T12:08:24.438283+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:24.438283+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2459,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1070}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26736,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:22.318339+0000","last_change":"2024-09-18T12:08:22.318339+0000","last_active":"2024-09-18T12:08:22.318339+0000","last_peered":"2024-09-18T12:08:22.318339+0000","last_clean":"2024-09-18T12:08:22.318339+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:22.318339+0000","last_undegraded":"2024-09-18T12:08:22.318339+0000","last_fullsized":"2024-09-18T12:08:22.318339+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37985,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":120111104,"data_stored":120100667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051758,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371828,"kb_used_data":61484,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94000012,"statfs":{"total":96636764160,"available":96256012288,"internally_reserved":0,"allocated":62959616,"data_stored":62647576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.47299999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.52500000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855521,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919460,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452380,"statfs":{"total":96636764160,"available":95695237120,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.41599999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.36899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.33300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62119936,"data_stored":62119936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:28.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:28.875 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:28.879 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:28.892 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-18T12:08:27.336774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37986,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1658664,"kb_used_data":119052,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281456856,"statfs":{"total":289910292480,"available":288211820544,"internally_reserved":0,"allocated":121909248,"data_stored":120947293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003400"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":24437,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:24.438283+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:24.438283+0000","last_peered":"2024-09-18T12:08:24.438283+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:24.438283+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2459,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1070}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26736,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:22.318339+0000","last_change":"2024-09-18T12:08:22.318339+0000","last_active":"2024-09-18T12:08:22.318339+0000","last_peered":"2024-09-18T12:08:22.318339+0000","last_clean":"2024-09-18T12:08:22.318339+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:22.318339+0000","last_undegraded":"2024-09-18T12:08:22.318339+0000","last_fullsized":"2024-09-18T12:08:22.318339+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37985,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":120111104,"data_stored":120100667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051758,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371828,"kb_used_data":61484,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94000012,"statfs":{"total":96636764160,"available":96256012288,"internally_reserved":0,"allocated":62959616,"data_stored":62647576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.47299999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.52500000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855521,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919460,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452380,"statfs":{"total":96636764160,"available":95695237120,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.41599999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.36899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.33300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":62119936,"data_stored":62119936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:28.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:08:28.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 24437, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:24.438283+0000', 'last_change': '2024-09-18T12:07:55.558532+0000', 'last_active': '2024-09-18T12:08:24.438283+0000', 'last_peered': '2024-09-18T12:08:24.438283+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:08:24.438283+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1072, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2459, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1070}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:28.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:28.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19628, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:55.656223+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:55.656223+0000', 'last_peered': '2024-09-18T12:07:55.656223+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:55.656223+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:28.894 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:29.257 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:29.337 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4608/44 objects degraded (10472.727%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":909,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1698471936,"bytes_avail":288211820544,"bytes_total":289910292480,"degraded_objects":4608,"degraded_total":44,"degraded_ratio":104.72727272727273,"recovering_objects_per_sec":78,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":940,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:29.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:29.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:29.638 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:29.641 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:29.655 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-18T12:08:29.337270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37986,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1655064,"kb_used_data":115452,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281460456,"statfs":{"total":289910292480,"available":288215506944,"internally_reserved":0,"allocated":118222848,"data_stored":117260893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003514"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":24437,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:24.438283+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:24.438283+0000","last_peered":"2024-09-18T12:08:24.438283+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:24.438283+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2459,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1070}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26748,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:28.368963+0000","last_change":"2024-09-18T12:08:28.368963+0000","last_active":"2024-09-18T12:08:28.368963+0000","last_peered":"2024-09-18T12:08:28.368963+0000","last_clean":"2024-09-18T12:08:28.368963+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:28.368963+0000","last_undegraded":"2024-09-18T12:08:28.368963+0000","last_fullsized":"2024-09-18T12:08:28.368963+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37985,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":116424704,"data_stored":116414267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051759,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368228,"kb_used_data":57884,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94003612,"statfs":{"total":96636764160,"available":96259698688,"internally_reserved":0,"allocated":59273216,"data_stored":58961176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38800000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855521,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919460,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452380,"statfs":{"total":96636764160,"available":95695237120,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.41599999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.36899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.33300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58433536,"data_stored":58433536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:32.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:32.981 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:32.983 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:32.995 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":506,"stamp":"2024-09-18T12:08:31.337959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38544,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1655064,"kb_used_data":115452,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281460456,"statfs":{"total":289910292480,"available":288215506944,"internally_reserved":0,"allocated":118222848,"data_stored":117260893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003509"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":25553,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:29.439400+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:29.439400+0000","last_peered":"2024-09-18T12:08:29.439400+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:29.439400+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26748,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:28.368963+0000","last_change":"2024-09-18T12:08:28.368963+0000","last_active":"2024-09-18T12:08:28.368963+0000","last_peered":"2024-09-18T12:08:28.368963+0000","last_clean":"2024-09-18T12:08:28.368963+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:28.368963+0000","last_undegraded":"2024-09-18T12:08:28.368963+0000","last_fullsized":"2024-09-18T12:08:28.368963+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38543,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":116424704,"data_stored":116414267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051759,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368228,"kb_used_data":57884,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94003612,"statfs":{"total":96636764160,"available":96259698688,"internally_reserved":0,"allocated":59273216,"data_stored":58961176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38800000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855521,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919460,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452380,"statfs":{"total":96636764160,"available":95695237120,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.41599999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.36899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.251},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.33200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58433536,"data_stored":58433536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:32.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:33.285 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:33.287 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:33.297 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":506,"stamp":"2024-09-18T12:08:31.337959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38544,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1655064,"kb_used_data":115452,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281460456,"statfs":{"total":289910292480,"available":288215506944,"internally_reserved":0,"allocated":118222848,"data_stored":117260893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003509"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":25553,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:29.439400+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:29.439400+0000","last_peered":"2024-09-18T12:08:29.439400+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:29.439400+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26748,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:28.368963+0000","last_change":"2024-09-18T12:08:28.368963+0000","last_active":"2024-09-18T12:08:28.368963+0000","last_peered":"2024-09-18T12:08:28.368963+0000","last_clean":"2024-09-18T12:08:28.368963+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:28.368963+0000","last_undegraded":"2024-09-18T12:08:28.368963+0000","last_fullsized":"2024-09-18T12:08:28.368963+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38543,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":116424704,"data_stored":116414267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051759,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368228,"kb_used_data":57884,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94003612,"statfs":{"total":96636764160,"available":96259698688,"internally_reserved":0,"allocated":59273216,"data_stored":58961176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38800000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855521,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919460,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452380,"statfs":{"total":96636764160,"available":95695237120,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.50700000000000001,"15min":0.44600000000000001},"min":{"1min":0.26200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.52100000000000002,"5min":8.6210000000000004,"15min":8.6210000000000004},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.80800000000000005,"15min":1.169},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":0.49399999999999999,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.41599999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":1.0029999999999999,"15min":1.286},"min":{"1min":0.29699999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.65700000000000003,"5min":11.41,"15min":13.737},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.57799999999999996,"15min":0.748},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.56399999999999995,"5min":6.7060000000000004,"15min":11.996},"last":0.36899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.251},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.33200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58433536,"data_stored":58433536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:33.299 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-18T12:08:33.299 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "43'19423", 'reported_seq': 25553, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:29.439400+0000', 'last_change': '2024-09-18T12:07:55.558532+0000', 'last_active': '2024-09-18T12:08:29.439400+0000', 'last_peered': '2024-09-18T12:08:29.439400+0000', 'last_clean': '2024-09-18T11:55:02.230453+0000', 'last_became_active': '2024-09-18T11:55:04.767458+0000', 'last_became_peered': '2024-09-18T11:55:04.767458+0000', 'last_unstale': '2024-09-18T12:08:29.439400+0000', 'last_undegraded': '2024-09-18T11:55:04.321126+0000', 'last_fullsized': '2024-09-18T11:55:04.249466+0000', 'mapping_epoch': 36, 'log_start': "39'9400", 'ondisk_log_start': "39'9400", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1234, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 514, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19421, 'num_write_kb': 38839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3017, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -512}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:33.299 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:33.299 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19628, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:07:55.656223+0000', 'last_change': '2024-09-18T11:55:04.794075+0000', 'last_active': '2024-09-18T12:07:55.656223+0000', 'last_peered': '2024-09-18T12:07:55.656223+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:07:55.656223+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3536, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:33.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:33.649 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:33.649 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4050/44 objects degraded (9204.545%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":913,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1694785536,"bytes_avail":288215506944,"bytes_total":289910292480,"degraded_objects":4050,"degraded_total":44,"degraded_ratio":92.045454545454547,"recovering_objects_per_sec":124,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1498,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:33.662 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:33.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:33.975 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:33.978 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:33.990 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2024-09-18T12:08:33.832747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38544,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1659224,"kb_used_data":115452,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281456296,"statfs":{"total":289910292480,"available":288211247104,"internally_reserved":0,"allocated":118222848,"data_stored":117260893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1053,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.497183"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26834,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.734610+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.734610+0000","last_peered":"2024-09-18T12:07:56.734610+0000","last_clean":"2024-09-18T12:07:56.734610+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.734610+0000","last_undegraded":"2024-09-18T12:07:56.734610+0000","last_fullsized":"2024-09-18T12:07:56.734610+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19608,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.668006+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:55.668006+0000","last_peered":"2024-09-18T12:07:55.668006+0000","last_clean":"2024-09-18T12:07:55.668006+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:55.668006+0000","last_undegraded":"2024-09-18T12:07:55.668006+0000","last_fullsized":"2024-09-18T12:07:55.668006+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19307,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667940+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:55.667940+0000","last_peered":"2024-09-18T12:07:55.667940+0000","last_clean":"2024-09-18T12:07:55.667940+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:55.667940+0000","last_undegraded":"2024-09-18T12:07:55.667940+0000","last_fullsized":"2024-09-18T12:07:55.667940+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19656,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655856+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:55.655856+0000","last_peered":"2024-09-18T12:07:55.655856+0000","last_clean":"2024-09-18T12:07:55.655856+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:55.655856+0000","last_undegraded":"2024-09-18T12:07:55.655856+0000","last_fullsized":"2024-09-18T12:07:55.655856+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":25553,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:29.439400+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:29.439400+0000","last_peered":"2024-09-18T12:08:29.439400+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:29.439400+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19845,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655431+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:55.655431+0000","last_peered":"2024-09-18T12:07:55.655431+0000","last_clean":"2024-09-18T12:07:55.655431+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:55.655431+0000","last_undegraded":"2024-09-18T12:07:55.655431+0000","last_fullsized":"2024-09-18T12:07:55.655431+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667499+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:55.667499+0000","last_peered":"2024-09-18T12:07:55.667499+0000","last_clean":"2024-09-18T12:07:55.667499+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:55.667499+0000","last_undegraded":"2024-09-18T12:07:55.667499+0000","last_fullsized":"2024-09-18T12:07:55.667499+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26621,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655747+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:55.655747+0000","last_peered":"2024-09-18T12:07:55.655747+0000","last_clean":"2024-09-18T12:07:55.655747+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:55.655747+0000","last_undegraded":"2024-09-18T12:07:55.655747+0000","last_fullsized":"2024-09-18T12:07:55.655747+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":109,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655469+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:55.655469+0000","last_peered":"2024-09-18T12:07:55.655469+0000","last_clean":"2024-09-18T12:07:55.655469+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:55.655469+0000","last_undegraded":"2024-09-18T12:07:55.655469+0000","last_fullsized":"2024-09-18T12:07:55.655469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26748,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:28.368963+0000","last_change":"2024-09-18T12:08:28.368963+0000","last_active":"2024-09-18T12:08:28.368963+0000","last_peered":"2024-09-18T12:08:28.368963+0000","last_clean":"2024-09-18T12:08:28.368963+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:28.368963+0000","last_undegraded":"2024-09-18T12:08:28.368963+0000","last_fullsized":"2024-09-18T12:08:28.368963+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26929,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667483+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:55.667483+0000","last_peered":"2024-09-18T12:07:55.667483+0000","last_clean":"2024-09-18T12:07:55.667483+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:55.667483+0000","last_undegraded":"2024-09-18T12:07:55.667483+0000","last_fullsized":"2024-09-18T12:07:55.667483+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T12:07:55.656223+0000","last_change":"2024-09-18T11:55:04.794075+0000","last_active":"2024-09-18T12:07:55.656223+0000","last_peered":"2024-09-18T12:07:55.656223+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:07:55.656223+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.667961+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:55.667961+0000","last_peered":"2024-09-18T12:07:55.667961+0000","last_clean":"2024-09-18T12:07:55.667961+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:55.667961+0000","last_undegraded":"2024-09-18T12:07:55.667961+0000","last_fullsized":"2024-09-18T12:07:55.667961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38543,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":116424704,"data_stored":116414267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051759,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368228,"kb_used_data":57884,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94003612,"statfs":{"total":96636764160,"available":96259698688,"internally_reserved":0,"allocated":59273216,"data_stored":58961176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.38800000000000001}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46000000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855522,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923620,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448220,"statfs":{"total":96636764160,"available":95690977280,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.55900000000000005}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.53700000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367376,"kb_used_data":57092,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004464,"statfs":{"total":96636764160,"available":96260571136,"internally_reserved":0,"allocated":58462208,"data_stored":58138990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.55700000000000005,"15min":0.94399999999999995},"min":{"1min":0.222,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.71399999999999997,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.251},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.54800000000000004,"15min":0.85399999999999998},"min":{"1min":0.255,"5min":0.189,"15min":0.189},"max":{"1min":0.57599999999999996,"5min":5.875,"15min":8.3460000000000001},"last":0.33200000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":58433536,"data_stored":58433536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:36.992 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:37.307 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:37.308 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:37.322 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-09-18T12:08:35.339047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38659,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1655632,"kb_used_data":111860,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281459888,"statfs":{"total":289910292480,"available":288214925312,"internally_reserved":0,"allocated":114544640,"data_stored":113577236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002273"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26835,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.735154+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.735154+0000","last_peered":"2024-09-18T12:07:56.735154+0000","last_clean":"2024-09-18T12:07:56.735154+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.735154+0000","last_undegraded":"2024-09-18T12:07:56.735154+0000","last_fullsized":"2024-09-18T12:07:56.735154+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19609,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.657118+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:56.657118+0000","last_peered":"2024-09-18T12:07:56.657118+0000","last_clean":"2024-09-18T12:07:56.657118+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:56.657118+0000","last_undegraded":"2024-09-18T12:07:56.657118+0000","last_fullsized":"2024-09-18T12:07:56.657118+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19767,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654437+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:07:55.654437+0000","last_peered":"2024-09-18T12:07:55.654437+0000","last_clean":"2024-09-18T12:07:55.654437+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:07:55.654437+0000","last_undegraded":"2024-09-18T12:07:55.654437+0000","last_fullsized":"2024-09-18T12:07:55.654437+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19308,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.656859+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:56.656859+0000","last_peered":"2024-09-18T12:07:56.656859+0000","last_clean":"2024-09-18T12:07:56.656859+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:56.656859+0000","last_undegraded":"2024-09-18T12:07:56.656859+0000","last_fullsized":"2024-09-18T12:07:56.656859+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19657,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.659203+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:56.659203+0000","last_peered":"2024-09-18T12:07:56.659203+0000","last_clean":"2024-09-18T12:07:56.659203+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:56.659203+0000","last_undegraded":"2024-09-18T12:07:56.659203+0000","last_fullsized":"2024-09-18T12:07:56.659203+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":25553,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:29.439400+0000","last_change":"2024-09-18T12:07:55.558532+0000","last_active":"2024-09-18T12:08:29.439400+0000","last_peered":"2024-09-18T12:08:29.439400+0000","last_clean":"2024-09-18T11:55:02.230453+0000","last_became_active":"2024-09-18T11:55:04.767458+0000","last_became_peered":"2024-09-18T11:55:04.767458+0000","last_unstale":"2024-09-18T12:08:29.439400+0000","last_undegraded":"2024-09-18T11:55:04.321126+0000","last_fullsized":"2024-09-18T11:55:04.249466+0000","mapping_epoch":36,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19477,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654504+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:07:55.654504+0000","last_peered":"2024-09-18T12:07:55.654504+0000","last_clean":"2024-09-18T12:07:55.654504+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:07:55.654504+0000","last_undegraded":"2024-09-18T12:07:55.654504+0000","last_fullsized":"2024-09-18T12:07:55.654504+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19846,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.658702+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:56.658702+0000","last_peered":"2024-09-18T12:07:56.658702+0000","last_clean":"2024-09-18T12:07:56.658702+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:56.658702+0000","last_undegraded":"2024-09-18T12:07:56.658702+0000","last_fullsized":"2024-09-18T12:07:56.658702+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654403+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:07:55.654403+0000","last_peered":"2024-09-18T12:07:55.654403+0000","last_clean":"2024-09-18T12:07:55.654403+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:07:55.654403+0000","last_undegraded":"2024-09-18T12:07:55.654403+0000","last_fullsized":"2024-09-18T12:07:55.654403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26551,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654549+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:07:55.654549+0000","last_peered":"2024-09-18T12:07:55.654549+0000","last_clean":"2024-09-18T12:07:55.654549+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:07:55.654549+0000","last_undegraded":"2024-09-18T12:07:55.654549+0000","last_fullsized":"2024-09-18T12:07:55.654549+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.656303+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:56.656303+0000","last_peered":"2024-09-18T12:07:56.656303+0000","last_clean":"2024-09-18T12:07:56.656303+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:56.656303+0000","last_undegraded":"2024-09-18T12:07:56.656303+0000","last_fullsized":"2024-09-18T12:07:56.656303+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26622,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.659835+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:56.659835+0000","last_peered":"2024-09-18T12:07:56.659835+0000","last_clean":"2024-09-18T12:07:56.659835+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:56.659835+0000","last_undegraded":"2024-09-18T12:07:56.659835+0000","last_fullsized":"2024-09-18T12:07:56.659835+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":110,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.658763+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:56.658763+0000","last_peered":"2024-09-18T12:07:56.658763+0000","last_clean":"2024-09-18T12:07:56.658763+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:56.658763+0000","last_undegraded":"2024-09-18T12:07:56.658763+0000","last_fullsized":"2024-09-18T12:07:56.658763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26755,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:32.333422+0000","last_change":"2024-09-18T12:08:32.333399+0000","last_active":"2024-09-18T12:08:32.333422+0000","last_peered":"2024-09-18T12:08:32.333422+0000","last_clean":"2024-09-18T12:08:32.333422+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:32.333422+0000","last_undegraded":"2024-09-18T12:08:32.333422+0000","last_fullsized":"2024-09-18T12:08:32.333422+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":89,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654431+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:07:55.654431+0000","last_peered":"2024-09-18T12:07:55.654431+0000","last_clean":"2024-09-18T12:07:55.654431+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:07:55.654431+0000","last_undegraded":"2024-09-18T12:07:55.654431+0000","last_fullsized":"2024-09-18T12:07:55.654431+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26930,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.657248+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:56.657248+0000","last_peered":"2024-09-18T12:07:56.657248+0000","last_clean":"2024-09-18T12:07:56.657248+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:56.657248+0000","last_undegraded":"2024-09-18T12:07:56.657248+0000","last_fullsized":"2024-09-18T12:07:56.657248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19860,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:34.443784+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:34.443784+0000","last_peered":"2024-09-18T12:08:34.443784+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:34.443784+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654686+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:07:55.654686+0000","last_peered":"2024-09-18T12:07:55.654686+0000","last_clean":"2024-09-18T12:07:55.654686+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:07:55.654686+0000","last_undegraded":"2024-09-18T12:07:55.654686+0000","last_fullsized":"2024-09-18T12:07:55.654686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26686,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.655099+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:07:55.655099+0000","last_peered":"2024-09-18T12:07:55.655099+0000","last_clean":"2024-09-18T12:07:55.655099+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:07:55.655099+0000","last_undegraded":"2024-09-18T12:07:55.655099+0000","last_fullsized":"2024-09-18T12:07:55.655099+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.657082+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:56.657082+0000","last_peered":"2024-09-18T12:07:56.657082+0000","last_clean":"2024-09-18T12:07:56.657082+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:56.657082+0000","last_undegraded":"2024-09-18T12:07:56.657082+0000","last_fullsized":"2024-09-18T12:07:56.657082+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19304,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654537+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:07:55.654537+0000","last_peered":"2024-09-18T12:07:55.654537+0000","last_clean":"2024-09-18T12:07:55.654537+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:07:55.654537+0000","last_undegraded":"2024-09-18T12:07:55.654537+0000","last_fullsized":"2024-09-18T12:07:55.654537+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":73,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654895+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:07:55.654895+0000","last_peered":"2024-09-18T12:07:55.654895+0000","last_clean":"2024-09-18T12:07:55.654895+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:07:55.654895+0000","last_undegraded":"2024-09-18T12:07:55.654895+0000","last_fullsized":"2024-09-18T12:07:55.654895+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":84,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654461+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:07:55.654461+0000","last_peered":"2024-09-18T12:07:55.654461+0000","last_clean":"2024-09-18T12:07:55.654461+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:07:55.654461+0000","last_undegraded":"2024-09-18T12:07:55.654461+0000","last_fullsized":"2024-09-18T12:07:55.654461+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19616,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-18T12:07:55.654528+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:07:55.654528+0000","last_peered":"2024-09-18T12:07:55.654528+0000","last_clean":"2024-09-18T12:07:55.654528+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:07:55.654528+0000","last_undegraded":"2024-09-18T12:07:55.654528+0000","last_fullsized":"2024-09-18T12:07:55.654528+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38658,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":112738304,"data_stored":112727867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051760,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364628,"kb_used_data":54284,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94007212,"statfs":{"total":96636764160,"available":96263385088,"internally_reserved":0,"allocated":55586816,"data_stored":55274776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.34100000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.32800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855522,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923620,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448220,"statfs":{"total":96636764160,"available":95690977280,"internally_reserved":0,"allocated":487424,"data_stored":160727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.55900000000000005}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.53700000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367384,"kb_used_data":57100,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004456,"statfs":{"total":96636764160,"available":96260562944,"internally_reserved":0,"allocated":58470400,"data_stored":58141733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.35799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.377},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:37.323 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:37.630 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:37.642 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-18T12:08:37.339489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39173,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1659808,"kb_used_data":111872,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281455712,"statfs":{"total":289910292480,"available":288210649088,"internally_reserved":0,"allocated":114556928,"data_stored":113582084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1187,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002218"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26835,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.735154+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.735154+0000","last_peered":"2024-09-18T12:07:56.735154+0000","last_clean":"2024-09-18T12:07:56.735154+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.735154+0000","last_undegraded":"2024-09-18T12:07:56.735154+0000","last_fullsized":"2024-09-18T12:07:56.735154+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19609,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.657118+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:56.657118+0000","last_peered":"2024-09-18T12:07:56.657118+0000","last_clean":"2024-09-18T12:07:56.657118+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:56.657118+0000","last_undegraded":"2024-09-18T12:07:56.657118+0000","last_fullsized":"2024-09-18T12:07:56.657118+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19308,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.656859+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:56.656859+0000","last_peered":"2024-09-18T12:07:56.656859+0000","last_clean":"2024-09-18T12:07:56.656859+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:56.656859+0000","last_undegraded":"2024-09-18T12:07:56.656859+0000","last_fullsized":"2024-09-18T12:07:56.656859+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19657,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.659203+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:56.659203+0000","last_peered":"2024-09-18T12:07:56.659203+0000","last_clean":"2024-09-18T12:07:56.659203+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:56.659203+0000","last_undegraded":"2024-09-18T12:07:56.659203+0000","last_fullsized":"2024-09-18T12:07:56.659203+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19846,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.658702+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:56.658702+0000","last_peered":"2024-09-18T12:07:56.658702+0000","last_clean":"2024-09-18T12:07:56.658702+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:56.658702+0000","last_undegraded":"2024-09-18T12:07:56.658702+0000","last_fullsized":"2024-09-18T12:07:56.658702+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.656303+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:56.656303+0000","last_peered":"2024-09-18T12:07:56.656303+0000","last_clean":"2024-09-18T12:07:56.656303+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:56.656303+0000","last_undegraded":"2024-09-18T12:07:56.656303+0000","last_fullsized":"2024-09-18T12:07:56.656303+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26622,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.659835+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:56.659835+0000","last_peered":"2024-09-18T12:07:56.659835+0000","last_clean":"2024-09-18T12:07:56.659835+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:56.659835+0000","last_undegraded":"2024-09-18T12:07:56.659835+0000","last_fullsized":"2024-09-18T12:07:56.659835+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":110,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.658763+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:56.658763+0000","last_peered":"2024-09-18T12:07:56.658763+0000","last_clean":"2024-09-18T12:07:56.658763+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:56.658763+0000","last_undegraded":"2024-09-18T12:07:56.658763+0000","last_fullsized":"2024-09-18T12:07:56.658763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26755,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:32.333422+0000","last_change":"2024-09-18T12:08:32.333399+0000","last_active":"2024-09-18T12:08:32.333422+0000","last_peered":"2024-09-18T12:08:32.333422+0000","last_clean":"2024-09-18T12:08:32.333422+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:32.333422+0000","last_undegraded":"2024-09-18T12:08:32.333422+0000","last_fullsized":"2024-09-18T12:08:32.333422+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26930,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.657248+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:56.657248+0000","last_peered":"2024-09-18T12:07:56.657248+0000","last_clean":"2024-09-18T12:07:56.657248+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:56.657248+0000","last_undegraded":"2024-09-18T12:07:56.657248+0000","last_fullsized":"2024-09-18T12:07:56.657248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19860,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:34.443784+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:34.443784+0000","last_peered":"2024-09-18T12:08:34.443784+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:34.443784+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.657082+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:56.657082+0000","last_peered":"2024-09-18T12:07:56.657082+0000","last_clean":"2024-09-18T12:07:56.657082+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:56.657082+0000","last_undegraded":"2024-09-18T12:07:56.657082+0000","last_fullsized":"2024-09-18T12:07:56.657082+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39172,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":112734208,"data_stored":112727250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051760,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364628,"kb_used_data":54284,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94007212,"statfs":{"total":96636764160,"available":96263385088,"internally_reserved":0,"allocated":55586816,"data_stored":55274776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.34100000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.32800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855523,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927796,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444044,"statfs":{"total":96636764160,"available":95686701056,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.35199999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.59599999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367384,"kb_used_data":57100,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004456,"statfs":{"total":96636764160,"available":96260562944,"internally_reserved":0,"allocated":58470400,"data_stored":58141733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.35799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.377},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:37.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:37.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 19860, 'reported_epoch': 58, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:34.443784+0000', 'last_change': '2024-09-18T12:08:33.346100+0000', 'last_active': '2024-09-18T12:08:34.443784+0000', 'last_peered': '2024-09-18T12:08:34.443784+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:08:34.443784+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3421, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 117, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3420}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:37.643 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:38.032 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:38.032 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3421/44 objects degraded (7775.000%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":918,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":59,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1699643392,"bytes_avail":288210649088,"bytes_total":289910292480,"degraded_objects":3421,"degraded_total":44,"degraded_ratio":77.75,"recovering_objects_per_sec":148,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1187,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:38.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:38.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:38.339 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:38.349 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-18T12:08:37.339489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39173,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1659808,"kb_used_data":111872,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281455712,"statfs":{"total":289910292480,"available":288210649088,"internally_reserved":0,"allocated":114556928,"data_stored":113582084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1187,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002218"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26835,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.735154+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:07:56.735154+0000","last_peered":"2024-09-18T12:07:56.735154+0000","last_clean":"2024-09-18T12:07:56.735154+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:07:56.735154+0000","last_undegraded":"2024-09-18T12:07:56.735154+0000","last_fullsized":"2024-09-18T12:07:56.735154+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19609,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.657118+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:07:56.657118+0000","last_peered":"2024-09-18T12:07:56.657118+0000","last_clean":"2024-09-18T12:07:56.657118+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:07:56.657118+0000","last_undegraded":"2024-09-18T12:07:56.657118+0000","last_fullsized":"2024-09-18T12:07:56.657118+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19308,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.656859+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:07:56.656859+0000","last_peered":"2024-09-18T12:07:56.656859+0000","last_clean":"2024-09-18T12:07:56.656859+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:07:56.656859+0000","last_undegraded":"2024-09-18T12:07:56.656859+0000","last_fullsized":"2024-09-18T12:07:56.656859+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19657,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.659203+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:07:56.659203+0000","last_peered":"2024-09-18T12:07:56.659203+0000","last_clean":"2024-09-18T12:07:56.659203+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:07:56.659203+0000","last_undegraded":"2024-09-18T12:07:56.659203+0000","last_fullsized":"2024-09-18T12:07:56.659203+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19846,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.658702+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:07:56.658702+0000","last_peered":"2024-09-18T12:07:56.658702+0000","last_clean":"2024-09-18T12:07:56.658702+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:07:56.658702+0000","last_undegraded":"2024-09-18T12:07:56.658702+0000","last_fullsized":"2024-09-18T12:07:56.658702+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.656303+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:07:56.656303+0000","last_peered":"2024-09-18T12:07:56.656303+0000","last_clean":"2024-09-18T12:07:56.656303+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:07:56.656303+0000","last_undegraded":"2024-09-18T12:07:56.656303+0000","last_fullsized":"2024-09-18T12:07:56.656303+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26622,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.659835+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:07:56.659835+0000","last_peered":"2024-09-18T12:07:56.659835+0000","last_clean":"2024-09-18T12:07:56.659835+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:07:56.659835+0000","last_undegraded":"2024-09-18T12:07:56.659835+0000","last_fullsized":"2024-09-18T12:07:56.659835+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":110,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.658763+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:07:56.658763+0000","last_peered":"2024-09-18T12:07:56.658763+0000","last_clean":"2024-09-18T12:07:56.658763+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:07:56.658763+0000","last_undegraded":"2024-09-18T12:07:56.658763+0000","last_fullsized":"2024-09-18T12:07:56.658763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26755,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:08:32.333422+0000","last_change":"2024-09-18T12:08:32.333399+0000","last_active":"2024-09-18T12:08:32.333422+0000","last_peered":"2024-09-18T12:08:32.333422+0000","last_clean":"2024-09-18T12:08:32.333422+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:32.333422+0000","last_undegraded":"2024-09-18T12:08:32.333422+0000","last_fullsized":"2024-09-18T12:08:32.333422+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26930,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.657248+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:07:56.657248+0000","last_peered":"2024-09-18T12:07:56.657248+0000","last_clean":"2024-09-18T12:07:56.657248+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:07:56.657248+0000","last_undegraded":"2024-09-18T12:07:56.657248+0000","last_fullsized":"2024-09-18T12:07:56.657248+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19860,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:34.443784+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:34.443784+0000","last_peered":"2024-09-18T12:08:34.443784+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:34.443784+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-18T12:07:56.657082+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:07:56.657082+0000","last_peered":"2024-09-18T12:07:56.657082+0000","last_clean":"2024-09-18T12:07:56.657082+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:07:56.657082+0000","last_undegraded":"2024-09-18T12:07:56.657082+0000","last_fullsized":"2024-09-18T12:07:56.657082+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39172,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":112734208,"data_stored":112727250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051760,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364628,"kb_used_data":54284,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94007212,"statfs":{"total":96636764160,"available":96263385088,"internally_reserved":0,"allocated":55586816,"data_stored":55274776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.34100000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.32800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855523,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927796,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444044,"statfs":{"total":96636764160,"available":95686701056,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.35199999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.59599999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367384,"kb_used_data":57100,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004456,"statfs":{"total":96636764160,"available":96260562944,"internally_reserved":0,"allocated":58470400,"data_stored":58141733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.35799999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.377},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:41.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:41.733 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:41.735 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:41.747 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-18T12:08:41.340383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39855,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1655884,"kb_used_data":107940,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281459636,"statfs":{"total":289910292480,"available":288214667264,"internally_reserved":0,"allocated":110530560,"data_stored":109539327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001832"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26762,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:37.281594+0000","last_change":"2024-09-18T12:08:37.281594+0000","last_active":"2024-09-18T12:08:37.281594+0000","last_peered":"2024-09-18T12:08:37.281594+0000","last_clean":"2024-09-18T12:08:37.281594+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:37.281594+0000","last_undegraded":"2024-09-18T12:08:37.281594+0000","last_fullsized":"2024-09-18T12:08:37.281594+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":21225,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:39.444424+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:39.444424+0000","last_peered":"2024-09-18T12:08:39.444424+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:39.444424+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39854,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":108560384,"data_stored":108553426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051761,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":360692,"kb_used_data":50344,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94011148,"statfs":{"total":96636764160,"available":96267415552,"internally_reserved":0,"allocated":51552256,"data_stored":51229297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.30399999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.373},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.44800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855523,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927796,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444044,"statfs":{"total":96636764160,"available":95686701056,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.35199999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.59599999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.30499999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50573312,"data_stored":50573312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:41.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:42.048 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:42.050 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:42.060 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-18T12:08:41.340383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39855,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1655884,"kb_used_data":107940,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281459636,"statfs":{"total":289910292480,"available":288214667264,"internally_reserved":0,"allocated":110530560,"data_stored":109539327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001832"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26762,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:37.281594+0000","last_change":"2024-09-18T12:08:37.281594+0000","last_active":"2024-09-18T12:08:37.281594+0000","last_peered":"2024-09-18T12:08:37.281594+0000","last_clean":"2024-09-18T12:08:37.281594+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:37.281594+0000","last_undegraded":"2024-09-18T12:08:37.281594+0000","last_fullsized":"2024-09-18T12:08:37.281594+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":21225,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:39.444424+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:39.444424+0000","last_peered":"2024-09-18T12:08:39.444424+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:39.444424+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39854,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":108560384,"data_stored":108553426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051761,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":360692,"kb_used_data":50344,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94011148,"statfs":{"total":96636764160,"available":96267415552,"internally_reserved":0,"allocated":51552256,"data_stored":51229297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.30399999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.373},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.44800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855523,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927796,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444044,"statfs":{"total":96636764160,"available":95686701056,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.35199999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.59599999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.30499999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50573312,"data_stored":50573312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:42.062 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:42.062 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 21225, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:39.444424+0000', 'last_change': '2024-09-18T12:08:33.346100+0000', 'last_active': '2024-09-18T12:08:39.444424+0000', 'last_peered': '2024-09-18T12:08:39.444424+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:08:39.444424+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2739, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 799, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2738}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:42.062 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:42.432 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:42.432 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2739/44 objects degraded (6225.000%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":922,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":59,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1695625216,"bytes_avail":288214667264,"bytes_total":289910292480,"degraded_objects":2739,"degraded_total":44,"degraded_ratio":62.25,"recovering_objects_per_sec":163,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:42.449 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:42.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:42.766 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:42.769 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:42.782 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-18T12:08:41.340383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39855,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1655884,"kb_used_data":107940,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281459636,"statfs":{"total":289910292480,"available":288214667264,"internally_reserved":0,"allocated":110530560,"data_stored":109539327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001832"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26762,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:37.281594+0000","last_change":"2024-09-18T12:08:37.281594+0000","last_active":"2024-09-18T12:08:37.281594+0000","last_peered":"2024-09-18T12:08:37.281594+0000","last_clean":"2024-09-18T12:08:37.281594+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:37.281594+0000","last_undegraded":"2024-09-18T12:08:37.281594+0000","last_fullsized":"2024-09-18T12:08:37.281594+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":21225,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:39.444424+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:39.444424+0000","last_peered":"2024-09-18T12:08:39.444424+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:39.444424+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39854,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":108560384,"data_stored":108553426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051761,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":360692,"kb_used_data":50344,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94011148,"statfs":{"total":96636764160,"available":96267415552,"internally_reserved":0,"allocated":51552256,"data_stored":51229297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.81000000000000005,"15min":0.71099999999999997},"min":{"1min":0.23100000000000001,"5min":0.185,"15min":0.153},"max":{"1min":0.52900000000000003,"5min":19.776,"15min":19.776},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.024,"15min":1.1619999999999999},"min":{"1min":0.25800000000000001,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.59599999999999997,"5min":26.702000000000002,"15min":26.702000000000002},"last":0.30399999999999999}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.373},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.44800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855523,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927796,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444044,"statfs":{"total":96636764160,"available":95686701056,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.35199999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.59599999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.30499999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50573312,"data_stored":50573312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:45.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:46.101 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:46.103 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:46.116 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-18T12:08:45.341287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40589,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1643356,"kb_used_data":103740,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281472164,"statfs":{"total":289910292480,"available":288227495936,"internally_reserved":0,"allocated":106229760,"data_stored":105238527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.505569"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26771,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:43.282957+0000","last_change":"2024-09-18T12:08:43.282957+0000","last_active":"2024-09-18T12:08:43.282957+0000","last_peered":"2024-09-18T12:08:43.282957+0000","last_clean":"2024-09-18T12:08:43.282957+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:43.282957+0000","last_undegraded":"2024-09-18T12:08:43.282957+0000","last_fullsized":"2024-09-18T12:08:43.282957+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":22693,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:44.448184+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:44.448184+0000","last_peered":"2024-09-18T12:08:44.448184+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:44.448184+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40588,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104259584,"data_stored":104252626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051762,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356488,"kb_used_data":46144,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94015352,"statfs":{"total":96636764160,"available":96271720448,"internally_reserved":0,"allocated":47251456,"data_stored":46928497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.317}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.441}]}]},{"osd":1,"up_from":35,"seq":150323855524,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919472,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452368,"statfs":{"total":96636764160,"available":95695224832,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.40400000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46272512,"data_stored":46272512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:46.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:46.404 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:46.408 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:46.422 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-18T12:08:45.341287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40589,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1643356,"kb_used_data":103740,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281472164,"statfs":{"total":289910292480,"available":288227495936,"internally_reserved":0,"allocated":106229760,"data_stored":105238527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.505569"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26771,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:43.282957+0000","last_change":"2024-09-18T12:08:43.282957+0000","last_active":"2024-09-18T12:08:43.282957+0000","last_peered":"2024-09-18T12:08:43.282957+0000","last_clean":"2024-09-18T12:08:43.282957+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:43.282957+0000","last_undegraded":"2024-09-18T12:08:43.282957+0000","last_fullsized":"2024-09-18T12:08:43.282957+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":22693,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:44.448184+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:44.448184+0000","last_peered":"2024-09-18T12:08:44.448184+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:44.448184+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40588,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104259584,"data_stored":104252626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051762,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356488,"kb_used_data":46144,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94015352,"statfs":{"total":96636764160,"available":96271720448,"internally_reserved":0,"allocated":47251456,"data_stored":46928497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.317}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.441}]}]},{"osd":1,"up_from":35,"seq":150323855524,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919472,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452368,"statfs":{"total":96636764160,"available":95695224832,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.40400000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46272512,"data_stored":46272512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:46.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:46.423 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 22693, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:44.448184+0000', 'last_change': '2024-09-18T12:08:33.346100+0000', 'last_active': '2024-09-18T12:08:44.448184+0000', 'last_peered': '2024-09-18T12:08:44.448184+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:08:44.448184+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1533, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2004}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:46.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:46.807 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2005/44 objects degraded (4556.818%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":926,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":59,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1682796544,"bytes_avail":288227495936,"bytes_total":289910292480,"degraded_objects":2005,"degraded_total":44,"degraded_ratio":45.56818181818182,"recovering_objects_per_sec":183,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:46.820 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:46.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:47.126 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:47.136 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-18T12:08:45.341287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40589,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1643356,"kb_used_data":103740,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281472164,"statfs":{"total":289910292480,"available":288227495936,"internally_reserved":0,"allocated":106229760,"data_stored":105238527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.505569"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26771,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:43.282957+0000","last_change":"2024-09-18T12:08:43.282957+0000","last_active":"2024-09-18T12:08:43.282957+0000","last_peered":"2024-09-18T12:08:43.282957+0000","last_clean":"2024-09-18T12:08:43.282957+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:43.282957+0000","last_undegraded":"2024-09-18T12:08:43.282957+0000","last_fullsized":"2024-09-18T12:08:43.282957+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":22693,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:44.448184+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:44.448184+0000","last_peered":"2024-09-18T12:08:44.448184+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:44.448184+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40588,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104259584,"data_stored":104252626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051762,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356488,"kb_used_data":46144,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94015352,"statfs":{"total":96636764160,"available":96271720448,"internally_reserved":0,"allocated":47251456,"data_stored":46928497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.317}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.441}]}]},{"osd":1,"up_from":35,"seq":150323855524,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919472,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452368,"statfs":{"total":96636764160,"available":95695224832,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.40400000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46272512,"data_stored":46272512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:50.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:50.472 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:50.474 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:50.484 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-18T12:08:49.342234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40589,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1643316,"kb_used_data":99540,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281472204,"statfs":{"total":289910292480,"available":288227536896,"internally_reserved":0,"allocated":101928960,"data_stored":100937727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002745"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26780,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:48.218170+0000","last_change":"2024-09-18T12:08:48.218170+0000","last_active":"2024-09-18T12:08:48.218170+0000","last_peered":"2024-09-18T12:08:48.218170+0000","last_clean":"2024-09-18T12:08:48.218170+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:48.218170+0000","last_undegraded":"2024-09-18T12:08:48.218170+0000","last_fullsized":"2024-09-18T12:08:48.218170+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":22693,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:44.448184+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:44.448184+0000","last_peered":"2024-09-18T12:08:44.448184+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:44.448184+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40588,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":99958784,"data_stored":99951826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051763,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352288,"kb_used_data":41944,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94019552,"statfs":{"total":96636764160,"available":96276021248,"internally_reserved":0,"allocated":42950656,"data_stored":42627697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.44}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.505},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46899999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855525,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923632,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448208,"statfs":{"total":96636764160,"available":95690964992,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.33900000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41971712,"data_stored":41971712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:50.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:50.795 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:50.797 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:50.808 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-18T12:08:49.342234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40589,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1643316,"kb_used_data":99540,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281472204,"statfs":{"total":289910292480,"available":288227536896,"internally_reserved":0,"allocated":101928960,"data_stored":100937727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002745"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26780,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:48.218170+0000","last_change":"2024-09-18T12:08:48.218170+0000","last_active":"2024-09-18T12:08:48.218170+0000","last_peered":"2024-09-18T12:08:48.218170+0000","last_clean":"2024-09-18T12:08:48.218170+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:48.218170+0000","last_undegraded":"2024-09-18T12:08:48.218170+0000","last_fullsized":"2024-09-18T12:08:48.218170+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":22693,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:44.448184+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:44.448184+0000","last_peered":"2024-09-18T12:08:44.448184+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:44.448184+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40588,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":99958784,"data_stored":99951826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051763,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352288,"kb_used_data":41944,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94019552,"statfs":{"total":96636764160,"available":96276021248,"internally_reserved":0,"allocated":42950656,"data_stored":42627697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.44}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.505},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46899999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855525,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923632,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448208,"statfs":{"total":96636764160,"available":95690964992,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.33900000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.36699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41971712,"data_stored":41971712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:50.809 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:50.809 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 22693, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:44.448184+0000', 'last_change': '2024-09-18T12:08:33.346100+0000', 'last_active': '2024-09-18T12:08:44.448184+0000', 'last_peered': '2024-09-18T12:08:44.448184+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:08:44.448184+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2005, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1533, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2004}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:50.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:51.185 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:51.185 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2005/44 objects degraded (4556.818%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":931,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":59,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1682755584,"bytes_avail":288227536896,"bytes_total":289910292480,"degraded_objects":2005,"degraded_total":44,"degraded_ratio":45.56818181818182,"recovering_objects_per_sec":117,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1416,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:51.202 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:51.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:51.509 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:51.513 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:51.521 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-09-18T12:08:51.342766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41097,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1643316,"kb_used_data":99540,"kb_used_omap":13,"kb_used_meta":1543730,"kb_avail":281472204,"statfs":{"total":289910292480,"available":288227536896,"internally_reserved":0,"allocated":101928960,"data_stored":100937727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1580780050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1924,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002837"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26780,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:48.218170+0000","last_change":"2024-09-18T12:08:48.218170+0000","last_active":"2024-09-18T12:08:48.218170+0000","last_peered":"2024-09-18T12:08:48.218170+0000","last_clean":"2024-09-18T12:08:48.218170+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:48.218170+0000","last_undegraded":"2024-09-18T12:08:48.218170+0000","last_fullsized":"2024-09-18T12:08:48.218170+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":23709,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:49.444382+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:49.444382+0000","last_peered":"2024-09-18T12:08:49.444382+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:49.444382+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41096,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":99958784,"data_stored":99951826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051763,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352288,"kb_used_data":41944,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94019552,"statfs":{"total":96636764160,"available":96276021248,"internally_reserved":0,"allocated":42950656,"data_stored":42627697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.44}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.505},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46899999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855525,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923632,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":923131,"kb_avail":93448208,"statfs":{"total":96636764160,"available":95690964992,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":945286662},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.33900000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.498}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41971712,"data_stored":41971712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:54.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:54.840 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:54.842 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:54.854 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2024-09-18T12:08:53.343256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41097,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1647476,"kb_used_data":99540,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281468044,"statfs":{"total":289910292480,"available":288223277056,"internally_reserved":0,"allocated":101928960,"data_stored":100937727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002872"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26780,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:48.218170+0000","last_change":"2024-09-18T12:08:48.218170+0000","last_active":"2024-09-18T12:08:48.218170+0000","last_peered":"2024-09-18T12:08:48.218170+0000","last_clean":"2024-09-18T12:08:48.218170+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:48.218170+0000","last_undegraded":"2024-09-18T12:08:48.218170+0000","last_fullsized":"2024-09-18T12:08:48.218170+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":23709,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:49.444382+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:49.444382+0000","last_peered":"2024-09-18T12:08:49.444382+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:49.444382+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41096,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":99958784,"data_stored":99951826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051763,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352288,"kb_used_data":41944,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94019552,"statfs":{"total":96636764160,"available":96276021248,"internally_reserved":0,"allocated":42950656,"data_stored":42627697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.44}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.505},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46899999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855526,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927792,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444048,"statfs":{"total":96636764160,"available":95686705152,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.308},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.503}]}]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.498}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41971712,"data_stored":41971712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:54.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:55.259 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:55.261 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:55.271 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2024-09-18T12:08:53.343256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41097,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1647476,"kb_used_data":99540,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281468044,"statfs":{"total":289910292480,"available":288223277056,"internally_reserved":0,"allocated":101928960,"data_stored":100937727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1242,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002872"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26780,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:48.218170+0000","last_change":"2024-09-18T12:08:48.218170+0000","last_active":"2024-09-18T12:08:48.218170+0000","last_peered":"2024-09-18T12:08:48.218170+0000","last_clean":"2024-09-18T12:08:48.218170+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:48.218170+0000","last_undegraded":"2024-09-18T12:08:48.218170+0000","last_fullsized":"2024-09-18T12:08:48.218170+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":23709,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:49.444382+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:49.444382+0000","last_peered":"2024-09-18T12:08:49.444382+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:49.444382+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41096,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":99958784,"data_stored":99951826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051763,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352288,"kb_used_data":41944,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94019552,"statfs":{"total":96636764160,"available":96276021248,"internally_reserved":0,"allocated":42950656,"data_stored":42627697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.44}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.505},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46899999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855526,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927792,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444048,"statfs":{"total":96636764160,"available":95686705152,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.308},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.503}]}]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.498}]},{"osd":2,"last update":"Wed Sep 18 12:07:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.82299999999999995,"15min":1.1299999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":11.071999999999999,"15min":13.773},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.82099999999999995,"15min":0.97899999999999998},"min":{"1min":0.28599999999999998,"5min":0.23699999999999999,"15min":0.17499999999999999},"max":{"1min":0.81999999999999995,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.46999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41971712,"data_stored":41971712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:55.273 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:55.273 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 23709, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:49.444382+0000', 'last_change': '2024-09-18T12:08:33.346100+0000', 'last_active': '2024-09-18T12:08:49.444382+0000', 'last_peered': '2024-09-18T12:08:49.444382+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:08:49.444382+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1497, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2041, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1496}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:55.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:08:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:55.677 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1497/44 objects degraded (3402.273%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":935,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":59,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1687015424,"bytes_avail":288223277056,"bytes_total":289910292480,"degraded_objects":1497,"degraded_total":44,"degraded_ratio":34.022727272727273,"recovering_objects_per_sec":103,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1242,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:08:55.691 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:08:55.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:56.009 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:56.011 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:56.021 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-09-18T12:08:55.343911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41595,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1643732,"kb_used_data":95796,"kb_used_omap":13,"kb_used_meta":1547890,"kb_avail":281471788,"statfs":{"total":289910292480,"available":288227110912,"internally_reserved":0,"allocated":98095104,"data_stored":97103871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1585039890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1740,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003107"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26786,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:53.196554+0000","last_change":"2024-09-18T12:08:53.196554+0000","last_active":"2024-09-18T12:08:53.196554+0000","last_peered":"2024-09-18T12:08:53.196554+0000","last_clean":"2024-09-18T12:08:53.196554+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:53.196554+0000","last_undegraded":"2024-09-18T12:08:53.196554+0000","last_fullsized":"2024-09-18T12:08:53.196554+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":24705,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:54.451636+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:54.451636+0000","last_peered":"2024-09-18T12:08:54.451636+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:54.451636+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2539,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41594,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96067584,"data_stored":96060626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051764,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348544,"kb_used_data":38200,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94023296,"statfs":{"total":96636764160,"available":96279855104,"internally_reserved":0,"allocated":39116800,"data_stored":38793841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.48299999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.38800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855526,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927792,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":927291,"kb_avail":93444048,"statfs":{"total":96636764160,"available":95686705152,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":949546502},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.308},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.503}]}]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.38},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.64000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38080512,"data_stored":38080512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:59.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:59.332 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:59.334 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:59.345 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2024-09-18T12:08:57.344354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41595,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1631252,"kb_used_data":95796,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281484268,"statfs":{"total":289910292480,"available":288239890432,"internally_reserved":0,"allocated":98095104,"data_stored":97103871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1006,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003066"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26786,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:53.196554+0000","last_change":"2024-09-18T12:08:53.196554+0000","last_active":"2024-09-18T12:08:53.196554+0000","last_peered":"2024-09-18T12:08:53.196554+0000","last_clean":"2024-09-18T12:08:53.196554+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:53.196554+0000","last_undegraded":"2024-09-18T12:08:53.196554+0000","last_fullsized":"2024-09-18T12:08:53.196554+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":24705,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:54.451636+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:54.451636+0000","last_peered":"2024-09-18T12:08:54.451636+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:54.451636+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2539,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41594,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96067584,"data_stored":96060626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051764,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348544,"kb_used_data":38200,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94023296,"statfs":{"total":96636764160,"available":96279855104,"internally_reserved":0,"allocated":39116800,"data_stored":38793841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.48299999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.38800000000000001}]}]},{"osd":1,"up_from":35,"seq":150323855527,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915312,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456528,"statfs":{"total":96636764160,"available":95699484672,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.48599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.58899999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.38},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.64000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38080512,"data_stored":38080512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:59.347 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:08:59.635 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:08:59.638 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:08:59.648 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-09-18T12:08:59.344871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41595,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1629180,"kb_used_data":93724,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281486340,"statfs":{"total":289910292480,"available":288242012160,"internally_reserved":0,"allocated":95973376,"data_stored":94982143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1006,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003128"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26792,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:56.161294+0000","last_change":"2024-09-18T12:08:56.161294+0000","last_active":"2024-09-18T12:08:56.161294+0000","last_peered":"2024-09-18T12:08:56.161294+0000","last_clean":"2024-09-18T12:08:56.161294+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:56.161294+0000","last_undegraded":"2024-09-18T12:08:56.161294+0000","last_fullsized":"2024-09-18T12:08:56.161294+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":24705,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:54.451636+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:54.451636+0000","last_peered":"2024-09-18T12:08:54.451636+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:54.451636+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2539,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41594,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94126080,"data_stored":94119122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346472,"kb_used_data":36128,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94025368,"statfs":{"total":96636764160,"available":96281976832,"internally_reserved":0,"allocated":36995072,"data_stored":36672113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.496},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.53200000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.59199999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855527,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915312,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456528,"statfs":{"total":96636764160,"available":95699484672,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.48599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.58899999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.38},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.64000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36139008,"data_stored":36139008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:08:59.650 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:08:59.650 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 24705, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:08:54.451636+0000', 'last_change': '2024-09-18T12:08:33.346100+0000', 'last_active': '2024-09-18T12:08:54.451636+0000', 'last_peered': '2024-09-18T12:08:54.451636+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:08:54.451636+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 999, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2539, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -998}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:08:59.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:09:00.062 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:00.062 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 999/44 objects degraded (2270.455%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":940,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":59,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1668280320,"bytes_avail":288242012160,"bytes_total":289910292480,"degraded_objects":999,"degraded_total":44,"degraded_ratio":22.704545454545453,"recovering_objects_per_sec":83,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1006,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:09:00.077 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:09:00.077 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:00.355 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:00.357 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:00.367 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-09-18T12:08:59.344871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41595,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1629180,"kb_used_data":93724,"kb_used_omap":13,"kb_used_meta":1535410,"kb_avail":281486340,"statfs":{"total":289910292480,"available":288242012160,"internally_reserved":0,"allocated":95973376,"data_stored":94982143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1572260370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1006,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003128"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26792,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:56.161294+0000","last_change":"2024-09-18T12:08:56.161294+0000","last_active":"2024-09-18T12:08:56.161294+0000","last_peered":"2024-09-18T12:08:56.161294+0000","last_clean":"2024-09-18T12:08:56.161294+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:56.161294+0000","last_undegraded":"2024-09-18T12:08:56.161294+0000","last_fullsized":"2024-09-18T12:08:56.161294+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":24705,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:08:54.451636+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:08:54.451636+0000","last_peered":"2024-09-18T12:08:54.451636+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:08:54.451636+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2539,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41594,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94126080,"data_stored":94119122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346472,"kb_used_data":36128,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94025368,"statfs":{"total":96636764160,"available":96281976832,"internally_reserved":0,"allocated":36995072,"data_stored":36672113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.496},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.53200000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.59199999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855527,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915312,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":914811,"kb_avail":93456528,"statfs":{"total":96636764160,"available":95699484672,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":936766982},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.48599999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.58899999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.38},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.64000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36139008,"data_stored":36139008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:03.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:03.665 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:03.667 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:03.677 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2024-09-18T12:09:03.346022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42424,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1633340,"kb_used_data":93724,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281482180,"statfs":{"total":289910292480,"available":288237752320,"internally_reserved":0,"allocated":95973376,"data_stored":94982143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002766"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26792,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:56.161294+0000","last_change":"2024-09-18T12:08:56.161294+0000","last_active":"2024-09-18T12:08:56.161294+0000","last_peered":"2024-09-18T12:08:56.161294+0000","last_clean":"2024-09-18T12:08:56.161294+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:56.161294+0000","last_undegraded":"2024-09-18T12:08:56.161294+0000","last_fullsized":"2024-09-18T12:08:56.161294+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":26363,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:09:01.301340+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:09:01.301340+0000","last_peered":"2024-09-18T12:09:01.301340+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:09:01.301340+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3368,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42423,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94126080,"data_stored":94119122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346472,"kb_used_data":36128,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94025368,"statfs":{"total":96636764160,"available":96281976832,"internally_reserved":0,"allocated":36995072,"data_stored":36672113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.496},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.53200000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.59199999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855528,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919472,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452368,"statfs":{"total":96636764160,"available":95695224832,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.53400000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247993,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36139008,"data_stored":36139008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:03.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:03.965 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:03.983 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2024-09-18T12:09:03.346022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42424,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1633340,"kb_used_data":93724,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281482180,"statfs":{"total":289910292480,"available":288237752320,"internally_reserved":0,"allocated":95973376,"data_stored":94982143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002766"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26792,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:56.161294+0000","last_change":"2024-09-18T12:08:56.161294+0000","last_active":"2024-09-18T12:08:56.161294+0000","last_peered":"2024-09-18T12:08:56.161294+0000","last_clean":"2024-09-18T12:08:56.161294+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:56.161294+0000","last_undegraded":"2024-09-18T12:08:56.161294+0000","last_fullsized":"2024-09-18T12:08:56.161294+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":26363,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:09:01.301340+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:09:01.301340+0000","last_peered":"2024-09-18T12:09:01.301340+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:09:01.301340+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3368,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42423,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94126080,"data_stored":94119122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346472,"kb_used_data":36128,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94025368,"statfs":{"total":96636764160,"available":96281976832,"internally_reserved":0,"allocated":36995072,"data_stored":36672113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.496},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.53200000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.59199999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855528,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919472,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452368,"statfs":{"total":96636764160,"available":95695224832,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.53400000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247993,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36139008,"data_stored":36139008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:03.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-18T12:09:03.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "43'19528", 'reported_seq': 26363, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T12:09:01.301340+0000', 'last_change': '2024-09-18T12:08:33.346100+0000', 'last_active': '2024-09-18T12:09:01.301340+0000', 'last_peered': '2024-09-18T12:09:01.301340+0000', 'last_clean': '2024-09-18T11:55:02.230686+0000', 'last_became_active': '2024-09-18T11:55:04.786441+0000', 'last_became_peered': '2024-09-18T11:55:04.786441+0000', 'last_unstale': '2024-09-18T12:09:01.301340+0000', 'last_undegraded': '2024-09-18T11:55:04.333113+0000', 'last_fullsized': '2024-09-18T11:55:04.252480+0000', 'mapping_epoch': 36, 'log_start': "39'9500", 'ondisk_log_start': "39'9500", 'created': 14, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'last_clean_scrub_stamp': '2024-09-18T11:53:41.166841+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 617, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 170, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19527, 'num_write_kb': 39053, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3368, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -169}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T12:09:03.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T12:09:04.308 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:04.308 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 170/44 objects degraded (386.364%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":944,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":61,"num_osds":3,"num_up_osds":3,"osd_up_since":1726660502,"num_in_osds":3,"osd_in_since":1726660496,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":22,"data_bytes":12359,"bytes_used":1672540160,"bytes_avail":288237752320,"bytes_total":289910292480,"degraded_objects":170,"degraded_total":44,"degraded_ratio":3.8636363636363638,"recovering_objects_per_sec":132,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1327,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T11:56:03.055831+0000","services":{}},"progress_events":{}} 2024-09-18T12:09:04.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-18T12:09:04.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:04.612 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:04.615 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:04.625 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-18T12:09:03.973327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12359,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295197,"num_write_kb":558419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42424,"num_bytes_recovered":35791484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1633340,"kb_used_data":93724,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281482180,"statfs":{"total":289910292480,"available":288237752320,"internally_reserved":0,"allocated":95973376,"data_stored":94982143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":829,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.629416"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26836,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836962+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:08:33.836962+0000","last_peered":"2024-09-18T12:08:33.836962+0000","last_clean":"2024-09-18T12:08:33.836962+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:08:33.836962+0000","last_undegraded":"2024-09-18T12:08:33.836962+0000","last_fullsized":"2024-09-18T12:08:33.836962+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19610,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840899+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:08:33.840899+0000","last_peered":"2024-09-18T12:08:33.840899+0000","last_clean":"2024-09-18T12:08:33.840899+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:08:33.840899+0000","last_undegraded":"2024-09-18T12:08:33.840899+0000","last_fullsized":"2024-09-18T12:08:33.840899+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19769,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836609+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:08:33.836609+0000","last_peered":"2024-09-18T12:08:33.836609+0000","last_clean":"2024-09-18T12:08:33.836609+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:08:33.836609+0000","last_undegraded":"2024-09-18T12:08:33.836609+0000","last_fullsized":"2024-09-18T12:08:33.836609+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19309,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840590+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:08:33.840590+0000","last_peered":"2024-09-18T12:08:33.840590+0000","last_clean":"2024-09-18T12:08:33.840590+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:08:33.840590+0000","last_undegraded":"2024-09-18T12:08:33.840590+0000","last_fullsized":"2024-09-18T12:08:33.840590+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19658,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835811+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:08:33.835811+0000","last_peered":"2024-09-18T12:08:33.835811+0000","last_clean":"2024-09-18T12:08:33.835811+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:08:33.835811+0000","last_undegraded":"2024-09-18T12:08:33.835811+0000","last_fullsized":"2024-09-18T12:08:33.835811+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19521,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:34.977085+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:08:34.977085+0000","last_peered":"2024-09-18T12:08:34.977085+0000","last_clean":"2024-09-18T12:08:34.977085+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:08:34.977085+0000","last_undegraded":"2024-09-18T12:08:34.977085+0000","last_fullsized":"2024-09-18T12:08:34.977085+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19479,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836195+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:08:33.836195+0000","last_peered":"2024-09-18T12:08:33.836195+0000","last_clean":"2024-09-18T12:08:33.836195+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:08:33.836195+0000","last_undegraded":"2024-09-18T12:08:33.836195+0000","last_fullsized":"2024-09-18T12:08:33.836195+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19847,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835166+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:08:33.835166+0000","last_peered":"2024-09-18T12:08:33.835166+0000","last_clean":"2024-09-18T12:08:33.835166+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:08:33.835166+0000","last_undegraded":"2024-09-18T12:08:33.835166+0000","last_fullsized":"2024-09-18T12:08:33.835166+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835377+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:08:33.835377+0000","last_peered":"2024-09-18T12:08:33.835377+0000","last_clean":"2024-09-18T12:08:33.835377+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:08:33.835377+0000","last_undegraded":"2024-09-18T12:08:33.835377+0000","last_fullsized":"2024-09-18T12:08:33.835377+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26553,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836855+0000","last_change":"2024-09-18T11:56:57.577998+0000","last_active":"2024-09-18T12:08:33.836855+0000","last_peered":"2024-09-18T12:08:33.836855+0000","last_clean":"2024-09-18T12:08:33.836855+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:08:33.836855+0000","last_undegraded":"2024-09-18T12:08:33.836855+0000","last_fullsized":"2024-09-18T12:08:33.836855+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":2160,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19403,"num_write_kb":38800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3483,"num_bytes_recovered":14249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840198+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:08:33.840198+0000","last_peered":"2024-09-18T12:08:33.840198+0000","last_clean":"2024-09-18T12:08:33.840198+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:08:33.840198+0000","last_undegraded":"2024-09-18T12:08:33.840198+0000","last_fullsized":"2024-09-18T12:08:33.840198+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26623,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835529+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:08:33.835529+0000","last_peered":"2024-09-18T12:08:33.835529+0000","last_clean":"2024-09-18T12:08:33.835529+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:08:33.835529+0000","last_undegraded":"2024-09-18T12:08:33.835529+0000","last_fullsized":"2024-09-18T12:08:33.835529+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":111,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.835091+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:08:33.835091+0000","last_peered":"2024-09-18T12:08:33.835091+0000","last_clean":"2024-09-18T12:08:33.835091+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:08:33.835091+0000","last_undegraded":"2024-09-18T12:08:33.835091+0000","last_fullsized":"2024-09-18T12:08:33.835091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26792,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-18T12:08:56.161294+0000","last_change":"2024-09-18T12:08:56.161294+0000","last_active":"2024-09-18T12:08:56.161294+0000","last_peered":"2024-09-18T12:08:56.161294+0000","last_clean":"2024-09-18T12:08:56.161294+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:08:56.161294+0000","last_undegraded":"2024-09-18T12:08:56.161294+0000","last_fullsized":"2024-09-18T12:08:56.161294+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19461,"num_write_kb":38919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":91,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836375+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:08:33.836375+0000","last_peered":"2024-09-18T12:08:33.836375+0000","last_clean":"2024-09-18T12:08:33.836375+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:08:33.836375+0000","last_undegraded":"2024-09-18T12:08:33.836375+0000","last_fullsized":"2024-09-18T12:08:33.836375+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26931,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.839036+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:08:33.839036+0000","last_peered":"2024-09-18T12:08:33.839036+0000","last_clean":"2024-09-18T12:08:33.839036+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:08:33.839036+0000","last_undegraded":"2024-09-18T12:08:33.839036+0000","last_fullsized":"2024-09-18T12:08:33.839036+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":26363,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T12:09:01.301340+0000","last_change":"2024-09-18T12:08:33.346100+0000","last_active":"2024-09-18T12:09:01.301340+0000","last_peered":"2024-09-18T12:09:01.301340+0000","last_clean":"2024-09-18T11:55:02.230686+0000","last_became_active":"2024-09-18T11:55:04.786441+0000","last_became_peered":"2024-09-18T11:55:04.786441+0000","last_unstale":"2024-09-18T12:09:01.301340+0000","last_undegraded":"2024-09-18T11:55:04.333113+0000","last_fullsized":"2024-09-18T11:55:04.252480+0000","mapping_epoch":36,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3368,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836523+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:08:33.836523+0000","last_peered":"2024-09-18T12:08:33.836523+0000","last_clean":"2024-09-18T12:08:33.836523+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:08:33.836523+0000","last_undegraded":"2024-09-18T12:08:33.836523+0000","last_fullsized":"2024-09-18T12:08:33.836523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26688,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836463+0000","last_change":"2024-09-18T11:59:16.843550+0000","last_active":"2024-09-18T12:08:33.836463+0000","last_peered":"2024-09-18T12:08:33.836463+0000","last_clean":"2024-09-18T12:08:33.836463+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:08:33.836463+0000","last_undegraded":"2024-09-18T12:08:33.836463+0000","last_fullsized":"2024-09-18T12:08:33.836463+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19519,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":14426112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.840790+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:08:33.840790+0000","last_peered":"2024-09-18T12:08:33.840790+0000","last_clean":"2024-09-18T12:08:33.840790+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:08:33.840790+0000","last_undegraded":"2024-09-18T12:08:33.840790+0000","last_fullsized":"2024-09-18T12:08:33.840790+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19306,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836675+0000","last_change":"2024-09-18T12:05:48.744541+0000","last_active":"2024-09-18T12:08:33.836675+0000","last_peered":"2024-09-18T12:08:33.836675+0000","last_clean":"2024-09-18T12:08:33.836675+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:08:33.836675+0000","last_undegraded":"2024-09-18T12:08:33.836675+0000","last_fullsized":"2024-09-18T12:08:33.836675+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19196,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3597,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836374+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:08:33.836374+0000","last_peered":"2024-09-18T12:08:33.836374+0000","last_clean":"2024-09-18T12:08:33.836374+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:08:33.836374+0000","last_undegraded":"2024-09-18T12:08:33.836374+0000","last_fullsized":"2024-09-18T12:08:33.836374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836380+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:08:33.836380+0000","last_peered":"2024-09-18T12:08:33.836380+0000","last_clean":"2024-09-18T12:08:33.836380+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:08:33.836380+0000","last_undegraded":"2024-09-18T12:08:33.836380+0000","last_fullsized":"2024-09-18T12:08:33.836380+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19618,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-18T12:08:33.836458+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:08:33.836458+0000","last_peered":"2024-09-18T12:08:33.836458+0000","last_clean":"2024-09-18T12:08:33.836458+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:08:33.836458+0000","last_undegraded":"2024-09-18T12:08:33.836458+0000","last_fullsized":"2024-09-18T12:08:33.836458+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":12340,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":295195,"num_write_kb":558417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42423,"num_bytes_recovered":35791465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94126080,"data_stored":94119122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":346472,"kb_used_data":36128,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94025368,"statfs":{"total":96636764160,"available":96281976832,"internally_reserved":0,"allocated":36995072,"data_stored":36672113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.496},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.53200000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.59199999999999997}]}]},{"osd":1,"up_from":35,"seq":150323855528,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919472,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452368,"statfs":{"total":96636764160,"available":95695224832,"internally_reserved":0,"allocated":499712,"data_stored":165575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.53400000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247993,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":367396,"kb_used_data":57108,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94004444,"statfs":{"total":96636764160,"available":96260550656,"internally_reserved":0,"allocated":58478592,"data_stored":58144455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57982976,"data_stored":57979497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36139008,"data_stored":36139008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:07.627 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:07.921 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:07.924 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:07.935 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-09-18T12:09:07.347207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6806,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217618,"num_write_kb":403278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28432,"num_bytes_recovered":7115388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1629704,"kb_used_data":90072,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281485816,"statfs":{"total":289910292480,"available":288241475584,"internally_reserved":0,"allocated":92233728,"data_stored":91209588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5553,"num_objects":-9,"num_object_clones":0,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-77579,"num_write_kb":-155141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-13163,"num_bytes_recovered":-28676096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002336"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26838,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974421+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:09:02.974421+0000","last_peered":"2024-09-18T12:09:02.974421+0000","last_clean":"2024-09-18T12:09:02.974421+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:09:02.974421+0000","last_undegraded":"2024-09-18T12:09:02.974421+0000","last_fullsized":"2024-09-18T12:09:02.974421+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19612,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978418+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:09:02.978418+0000","last_peered":"2024-09-18T12:09:02.978418+0000","last_clean":"2024-09-18T12:09:02.978418+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:09:02.978418+0000","last_undegraded":"2024-09-18T12:09:02.978418+0000","last_fullsized":"2024-09-18T12:09:02.978418+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19771,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974549+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:09:02.974549+0000","last_peered":"2024-09-18T12:09:02.974549+0000","last_clean":"2024-09-18T12:09:02.974549+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:09:02.974549+0000","last_undegraded":"2024-09-18T12:09:02.974549+0000","last_fullsized":"2024-09-18T12:09:02.974549+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19311,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978306+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:09:02.978306+0000","last_peered":"2024-09-18T12:09:02.978306+0000","last_clean":"2024-09-18T12:09:02.978306+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:09:02.978306+0000","last_undegraded":"2024-09-18T12:09:02.978306+0000","last_fullsized":"2024-09-18T12:09:02.978306+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19660,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973515+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:09:02.973515+0000","last_peered":"2024-09-18T12:09:02.973515+0000","last_clean":"2024-09-18T12:09:02.973515+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:09:02.973515+0000","last_undegraded":"2024-09-18T12:09:02.973515+0000","last_fullsized":"2024-09-18T12:09:02.973515+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19523,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974053+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:09:02.974053+0000","last_peered":"2024-09-18T12:09:02.974053+0000","last_clean":"2024-09-18T12:09:02.974053+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:09:02.974053+0000","last_undegraded":"2024-09-18T12:09:02.974053+0000","last_fullsized":"2024-09-18T12:09:02.974053+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19481,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973774+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:09:02.973774+0000","last_peered":"2024-09-18T12:09:02.973774+0000","last_clean":"2024-09-18T12:09:02.973774+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:09:02.973774+0000","last_undegraded":"2024-09-18T12:09:02.973774+0000","last_fullsized":"2024-09-18T12:09:02.973774+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973147+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:09:02.973147+0000","last_peered":"2024-09-18T12:09:02.973147+0000","last_clean":"2024-09-18T12:09:02.973147+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:09:02.973147+0000","last_undegraded":"2024-09-18T12:09:02.973147+0000","last_fullsized":"2024-09-18T12:09:02.973147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26565,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.550867+0000","last_change":"2024-09-18T12:09:03.550799+0000","last_active":"2024-09-18T12:09:03.550867+0000","last_peered":"2024-09-18T12:09:03.550867+0000","last_clean":"2024-09-18T12:09:03.550867+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:09:03.550867+0000","last_undegraded":"2024-09-18T12:09:03.550867+0000","last_fullsized":"2024-09-18T12:09:03.550867+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19405","last_scrub_stamp":"2024-09-18T12:09:03.550646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.550646+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:36:34.684302+0000","scrub_duration":0.021530328000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978072+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:09:02.978072+0000","last_peered":"2024-09-18T12:09:02.978072+0000","last_clean":"2024-09-18T12:09:02.978072+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:09:02.978072+0000","last_undegraded":"2024-09-18T12:09:02.978072+0000","last_fullsized":"2024-09-18T12:09:02.978072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26625,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973377+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:09:02.973377+0000","last_peered":"2024-09-18T12:09:02.973377+0000","last_clean":"2024-09-18T12:09:02.973377+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:09:02.973377+0000","last_undegraded":"2024-09-18T12:09:02.973377+0000","last_fullsized":"2024-09-18T12:09:02.973377+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":113,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26807,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.055835+0000","last_change":"2024-09-18T12:09:03.055754+0000","last_active":"2024-09-18T12:09:03.055835+0000","last_peered":"2024-09-18T12:09:03.055835+0000","last_clean":"2024-09-18T12:09:03.055835+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:09:03.055835+0000","last_undegraded":"2024-09-18T12:09:03.055835+0000","last_fullsized":"2024-09-18T12:09:03.055835+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19463","last_scrub_stamp":"2024-09-18T12:09:03.055575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.055575+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:38:03.335605+0000","scrub_duration":0.016902436999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":93,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973654+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:09:02.973654+0000","last_peered":"2024-09-18T12:09:02.973654+0000","last_clean":"2024-09-18T12:09:02.973654+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:09:02.973654+0000","last_undegraded":"2024-09-18T12:09:02.973654+0000","last_fullsized":"2024-09-18T12:09:02.973654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26933,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978133+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:09:02.978133+0000","last_peered":"2024-09-18T12:09:02.978133+0000","last_clean":"2024-09-18T12:09:02.978133+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:09:02.978133+0000","last_undegraded":"2024-09-18T12:09:02.978133+0000","last_fullsized":"2024-09-18T12:09:02.978133+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19627,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.058553+0000","last_change":"2024-09-18T12:09:04.058553+0000","last_active":"2024-09-18T12:09:04.058553+0000","last_peered":"2024-09-18T12:09:04.058553+0000","last_clean":"2024-09-18T12:09:04.058553+0000","last_became_active":"2024-09-18T12:09:03.980052+0000","last_became_peered":"2024-09-18T12:09:03.980052+0000","last_unstale":"2024-09-18T12:09:04.058553+0000","last_undegraded":"2024-09-18T12:09:04.058553+0000","last_fullsized":"2024-09-18T12:09:04.058553+0000","mapping_epoch":60,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3538,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974525+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:09:02.974525+0000","last_peered":"2024-09-18T12:09:02.974525+0000","last_clean":"2024-09-18T12:09:02.974525+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:09:02.974525+0000","last_undegraded":"2024-09-18T12:09:02.974525+0000","last_fullsized":"2024-09-18T12:09:02.974525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26700,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.520719+0000","last_change":"2024-09-18T12:09:04.520696+0000","last_active":"2024-09-18T12:09:04.520719+0000","last_peered":"2024-09-18T12:09:04.520719+0000","last_clean":"2024-09-18T12:09:04.520719+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:09:04.520719+0000","last_undegraded":"2024-09-18T12:09:04.520719+0000","last_fullsized":"2024-09-18T12:09:04.520719+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19522","last_scrub_stamp":"2024-09-18T12:09:04.520600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:04.520600+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:36:19.536586+0000","scrub_duration":0.018151014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978460+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:09:02.978460+0000","last_peered":"2024-09-18T12:09:02.978460+0000","last_clean":"2024-09-18T12:09:02.978460+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:09:02.978460+0000","last_undegraded":"2024-09-18T12:09:02.978460+0000","last_fullsized":"2024-09-18T12:09:02.978460+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19318,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:06.520625+0000","last_change":"2024-09-18T12:09:06.520583+0000","last_active":"2024-09-18T12:09:06.520625+0000","last_peered":"2024-09-18T12:09:06.520625+0000","last_clean":"2024-09-18T12:09:06.520625+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:09:06.520625+0000","last_undegraded":"2024-09-18T12:09:06.520625+0000","last_fullsized":"2024-09-18T12:09:06.520625+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19198","last_scrub_stamp":"2024-09-18T12:09:06.520451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:06.520451+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:12:09.759949+0000","scrub_duration":0.020014351999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974547+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:09:02.974547+0000","last_peered":"2024-09-18T12:09:02.974547+0000","last_clean":"2024-09-18T12:09:02.974547+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:09:02.974547+0000","last_undegraded":"2024-09-18T12:09:02.974547+0000","last_fullsized":"2024-09-18T12:09:02.974547+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":88,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974620+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:09:02.974620+0000","last_peered":"2024-09-18T12:09:02.974620+0000","last_clean":"2024-09-18T12:09:02.974620+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:09:02.974620+0000","last_undegraded":"2024-09-18T12:09:02.974620+0000","last_fullsized":"2024-09-18T12:09:02.974620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19620,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974474+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:09:02.974474+0000","last_peered":"2024-09-18T12:09:02.974474+0000","last_clean":"2024-09-18T12:09:02.974474+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:09:02.974474+0000","last_undegraded":"2024-09-18T12:09:02.974474+0000","last_fullsized":"2024-09-18T12:09:02.974474+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6787,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217616,"num_write_kb":403276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28431,"num_bytes_recovered":7115369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90214400,"data_stored":90207442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344220,"kb_used_data":33868,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94027620,"statfs":{"total":96636764160,"available":96284282880,"internally_reserved":0,"allocated":34680832,"data_stored":34346904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.35099999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855529,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919496,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452344,"statfs":{"total":96636764160,"available":95695200256,"internally_reserved":0,"allocated":516096,"data_stored":170990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.45800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.61599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247994,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365988,"kb_used_data":55700,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94005852,"statfs":{"total":96636764160,"available":96261992448,"internally_reserved":0,"allocated":57036800,"data_stored":56691694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.52500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56033280,"data_stored":56029801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34177024,"data_stored":34177024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:07.937 INFO:tasks.rep_lost_unfound_delete.ceph_manager:clean! 2024-09-18T12:09:07.937 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for clean 2024-09-18T12:09:07.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:08.230 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:08.233 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:08.244 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-09-18T12:09:07.347207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6806,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217618,"num_write_kb":403278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28432,"num_bytes_recovered":7115388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1629704,"kb_used_data":90072,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281485816,"statfs":{"total":289910292480,"available":288241475584,"internally_reserved":0,"allocated":92233728,"data_stored":91209588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5553,"num_objects":-9,"num_object_clones":0,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-77579,"num_write_kb":-155141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-13163,"num_bytes_recovered":-28676096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002336"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26838,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974421+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:09:02.974421+0000","last_peered":"2024-09-18T12:09:02.974421+0000","last_clean":"2024-09-18T12:09:02.974421+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:09:02.974421+0000","last_undegraded":"2024-09-18T12:09:02.974421+0000","last_fullsized":"2024-09-18T12:09:02.974421+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19612,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978418+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:09:02.978418+0000","last_peered":"2024-09-18T12:09:02.978418+0000","last_clean":"2024-09-18T12:09:02.978418+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:09:02.978418+0000","last_undegraded":"2024-09-18T12:09:02.978418+0000","last_fullsized":"2024-09-18T12:09:02.978418+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19771,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974549+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:09:02.974549+0000","last_peered":"2024-09-18T12:09:02.974549+0000","last_clean":"2024-09-18T12:09:02.974549+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:09:02.974549+0000","last_undegraded":"2024-09-18T12:09:02.974549+0000","last_fullsized":"2024-09-18T12:09:02.974549+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19311,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978306+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:09:02.978306+0000","last_peered":"2024-09-18T12:09:02.978306+0000","last_clean":"2024-09-18T12:09:02.978306+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:09:02.978306+0000","last_undegraded":"2024-09-18T12:09:02.978306+0000","last_fullsized":"2024-09-18T12:09:02.978306+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19660,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973515+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:09:02.973515+0000","last_peered":"2024-09-18T12:09:02.973515+0000","last_clean":"2024-09-18T12:09:02.973515+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:09:02.973515+0000","last_undegraded":"2024-09-18T12:09:02.973515+0000","last_fullsized":"2024-09-18T12:09:02.973515+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19523,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974053+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:09:02.974053+0000","last_peered":"2024-09-18T12:09:02.974053+0000","last_clean":"2024-09-18T12:09:02.974053+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:09:02.974053+0000","last_undegraded":"2024-09-18T12:09:02.974053+0000","last_fullsized":"2024-09-18T12:09:02.974053+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19481,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973774+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:09:02.973774+0000","last_peered":"2024-09-18T12:09:02.973774+0000","last_clean":"2024-09-18T12:09:02.973774+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:09:02.973774+0000","last_undegraded":"2024-09-18T12:09:02.973774+0000","last_fullsized":"2024-09-18T12:09:02.973774+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973147+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:09:02.973147+0000","last_peered":"2024-09-18T12:09:02.973147+0000","last_clean":"2024-09-18T12:09:02.973147+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:09:02.973147+0000","last_undegraded":"2024-09-18T12:09:02.973147+0000","last_fullsized":"2024-09-18T12:09:02.973147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26565,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.550867+0000","last_change":"2024-09-18T12:09:03.550799+0000","last_active":"2024-09-18T12:09:03.550867+0000","last_peered":"2024-09-18T12:09:03.550867+0000","last_clean":"2024-09-18T12:09:03.550867+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:09:03.550867+0000","last_undegraded":"2024-09-18T12:09:03.550867+0000","last_fullsized":"2024-09-18T12:09:03.550867+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19405","last_scrub_stamp":"2024-09-18T12:09:03.550646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.550646+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:36:34.684302+0000","scrub_duration":0.021530328000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978072+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:09:02.978072+0000","last_peered":"2024-09-18T12:09:02.978072+0000","last_clean":"2024-09-18T12:09:02.978072+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:09:02.978072+0000","last_undegraded":"2024-09-18T12:09:02.978072+0000","last_fullsized":"2024-09-18T12:09:02.978072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26625,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973377+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:09:02.973377+0000","last_peered":"2024-09-18T12:09:02.973377+0000","last_clean":"2024-09-18T12:09:02.973377+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:09:02.973377+0000","last_undegraded":"2024-09-18T12:09:02.973377+0000","last_fullsized":"2024-09-18T12:09:02.973377+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":113,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26807,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.055835+0000","last_change":"2024-09-18T12:09:03.055754+0000","last_active":"2024-09-18T12:09:03.055835+0000","last_peered":"2024-09-18T12:09:03.055835+0000","last_clean":"2024-09-18T12:09:03.055835+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:09:03.055835+0000","last_undegraded":"2024-09-18T12:09:03.055835+0000","last_fullsized":"2024-09-18T12:09:03.055835+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19463","last_scrub_stamp":"2024-09-18T12:09:03.055575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.055575+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:38:03.335605+0000","scrub_duration":0.016902436999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":93,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973654+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:09:02.973654+0000","last_peered":"2024-09-18T12:09:02.973654+0000","last_clean":"2024-09-18T12:09:02.973654+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:09:02.973654+0000","last_undegraded":"2024-09-18T12:09:02.973654+0000","last_fullsized":"2024-09-18T12:09:02.973654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26933,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978133+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:09:02.978133+0000","last_peered":"2024-09-18T12:09:02.978133+0000","last_clean":"2024-09-18T12:09:02.978133+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:09:02.978133+0000","last_undegraded":"2024-09-18T12:09:02.978133+0000","last_fullsized":"2024-09-18T12:09:02.978133+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19627,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.058553+0000","last_change":"2024-09-18T12:09:04.058553+0000","last_active":"2024-09-18T12:09:04.058553+0000","last_peered":"2024-09-18T12:09:04.058553+0000","last_clean":"2024-09-18T12:09:04.058553+0000","last_became_active":"2024-09-18T12:09:03.980052+0000","last_became_peered":"2024-09-18T12:09:03.980052+0000","last_unstale":"2024-09-18T12:09:04.058553+0000","last_undegraded":"2024-09-18T12:09:04.058553+0000","last_fullsized":"2024-09-18T12:09:04.058553+0000","mapping_epoch":60,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3538,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974525+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:09:02.974525+0000","last_peered":"2024-09-18T12:09:02.974525+0000","last_clean":"2024-09-18T12:09:02.974525+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:09:02.974525+0000","last_undegraded":"2024-09-18T12:09:02.974525+0000","last_fullsized":"2024-09-18T12:09:02.974525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26700,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.520719+0000","last_change":"2024-09-18T12:09:04.520696+0000","last_active":"2024-09-18T12:09:04.520719+0000","last_peered":"2024-09-18T12:09:04.520719+0000","last_clean":"2024-09-18T12:09:04.520719+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:09:04.520719+0000","last_undegraded":"2024-09-18T12:09:04.520719+0000","last_fullsized":"2024-09-18T12:09:04.520719+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19522","last_scrub_stamp":"2024-09-18T12:09:04.520600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:04.520600+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:36:19.536586+0000","scrub_duration":0.018151014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978460+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:09:02.978460+0000","last_peered":"2024-09-18T12:09:02.978460+0000","last_clean":"2024-09-18T12:09:02.978460+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:09:02.978460+0000","last_undegraded":"2024-09-18T12:09:02.978460+0000","last_fullsized":"2024-09-18T12:09:02.978460+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19318,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:06.520625+0000","last_change":"2024-09-18T12:09:06.520583+0000","last_active":"2024-09-18T12:09:06.520625+0000","last_peered":"2024-09-18T12:09:06.520625+0000","last_clean":"2024-09-18T12:09:06.520625+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:09:06.520625+0000","last_undegraded":"2024-09-18T12:09:06.520625+0000","last_fullsized":"2024-09-18T12:09:06.520625+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19198","last_scrub_stamp":"2024-09-18T12:09:06.520451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:06.520451+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:12:09.759949+0000","scrub_duration":0.020014351999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974547+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:09:02.974547+0000","last_peered":"2024-09-18T12:09:02.974547+0000","last_clean":"2024-09-18T12:09:02.974547+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:09:02.974547+0000","last_undegraded":"2024-09-18T12:09:02.974547+0000","last_fullsized":"2024-09-18T12:09:02.974547+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":88,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974620+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:09:02.974620+0000","last_peered":"2024-09-18T12:09:02.974620+0000","last_clean":"2024-09-18T12:09:02.974620+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:09:02.974620+0000","last_undegraded":"2024-09-18T12:09:02.974620+0000","last_fullsized":"2024-09-18T12:09:02.974620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19620,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974474+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:09:02.974474+0000","last_peered":"2024-09-18T12:09:02.974474+0000","last_clean":"2024-09-18T12:09:02.974474+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:09:02.974474+0000","last_undegraded":"2024-09-18T12:09:02.974474+0000","last_fullsized":"2024-09-18T12:09:02.974474+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6787,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217616,"num_write_kb":403276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28431,"num_bytes_recovered":7115369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90214400,"data_stored":90207442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344220,"kb_used_data":33868,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94027620,"statfs":{"total":96636764160,"available":96284282880,"internally_reserved":0,"allocated":34680832,"data_stored":34346904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.35099999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855529,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919496,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452344,"statfs":{"total":96636764160,"available":95695200256,"internally_reserved":0,"allocated":516096,"data_stored":170990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.45800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.61599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247994,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365988,"kb_used_data":55700,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94005852,"statfs":{"total":96636764160,"available":96261992448,"internally_reserved":0,"allocated":57036800,"data_stored":56691694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.52500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56033280,"data_stored":56029801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34177024,"data_stored":34177024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:08.246 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:08.539 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:08.541 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:08.557 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-09-18T12:09:07.347207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6806,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217618,"num_write_kb":403278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28432,"num_bytes_recovered":7115388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1629704,"kb_used_data":90072,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281485816,"statfs":{"total":289910292480,"available":288241475584,"internally_reserved":0,"allocated":92233728,"data_stored":91209588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5553,"num_objects":-9,"num_object_clones":0,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-77579,"num_write_kb":-155141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-13163,"num_bytes_recovered":-28676096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002336"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26838,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974421+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:09:02.974421+0000","last_peered":"2024-09-18T12:09:02.974421+0000","last_clean":"2024-09-18T12:09:02.974421+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:09:02.974421+0000","last_undegraded":"2024-09-18T12:09:02.974421+0000","last_fullsized":"2024-09-18T12:09:02.974421+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19612,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978418+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:09:02.978418+0000","last_peered":"2024-09-18T12:09:02.978418+0000","last_clean":"2024-09-18T12:09:02.978418+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:09:02.978418+0000","last_undegraded":"2024-09-18T12:09:02.978418+0000","last_fullsized":"2024-09-18T12:09:02.978418+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19771,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974549+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:09:02.974549+0000","last_peered":"2024-09-18T12:09:02.974549+0000","last_clean":"2024-09-18T12:09:02.974549+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:09:02.974549+0000","last_undegraded":"2024-09-18T12:09:02.974549+0000","last_fullsized":"2024-09-18T12:09:02.974549+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19311,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978306+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:09:02.978306+0000","last_peered":"2024-09-18T12:09:02.978306+0000","last_clean":"2024-09-18T12:09:02.978306+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:09:02.978306+0000","last_undegraded":"2024-09-18T12:09:02.978306+0000","last_fullsized":"2024-09-18T12:09:02.978306+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19660,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973515+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:09:02.973515+0000","last_peered":"2024-09-18T12:09:02.973515+0000","last_clean":"2024-09-18T12:09:02.973515+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:09:02.973515+0000","last_undegraded":"2024-09-18T12:09:02.973515+0000","last_fullsized":"2024-09-18T12:09:02.973515+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19523,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974053+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:09:02.974053+0000","last_peered":"2024-09-18T12:09:02.974053+0000","last_clean":"2024-09-18T12:09:02.974053+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:09:02.974053+0000","last_undegraded":"2024-09-18T12:09:02.974053+0000","last_fullsized":"2024-09-18T12:09:02.974053+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19481,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973774+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:09:02.973774+0000","last_peered":"2024-09-18T12:09:02.973774+0000","last_clean":"2024-09-18T12:09:02.973774+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:09:02.973774+0000","last_undegraded":"2024-09-18T12:09:02.973774+0000","last_fullsized":"2024-09-18T12:09:02.973774+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973147+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:09:02.973147+0000","last_peered":"2024-09-18T12:09:02.973147+0000","last_clean":"2024-09-18T12:09:02.973147+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:09:02.973147+0000","last_undegraded":"2024-09-18T12:09:02.973147+0000","last_fullsized":"2024-09-18T12:09:02.973147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26565,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.550867+0000","last_change":"2024-09-18T12:09:03.550799+0000","last_active":"2024-09-18T12:09:03.550867+0000","last_peered":"2024-09-18T12:09:03.550867+0000","last_clean":"2024-09-18T12:09:03.550867+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:09:03.550867+0000","last_undegraded":"2024-09-18T12:09:03.550867+0000","last_fullsized":"2024-09-18T12:09:03.550867+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19405","last_scrub_stamp":"2024-09-18T12:09:03.550646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.550646+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:36:34.684302+0000","scrub_duration":0.021530328000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978072+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:09:02.978072+0000","last_peered":"2024-09-18T12:09:02.978072+0000","last_clean":"2024-09-18T12:09:02.978072+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:09:02.978072+0000","last_undegraded":"2024-09-18T12:09:02.978072+0000","last_fullsized":"2024-09-18T12:09:02.978072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26625,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973377+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:09:02.973377+0000","last_peered":"2024-09-18T12:09:02.973377+0000","last_clean":"2024-09-18T12:09:02.973377+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:09:02.973377+0000","last_undegraded":"2024-09-18T12:09:02.973377+0000","last_fullsized":"2024-09-18T12:09:02.973377+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":113,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26807,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.055835+0000","last_change":"2024-09-18T12:09:03.055754+0000","last_active":"2024-09-18T12:09:03.055835+0000","last_peered":"2024-09-18T12:09:03.055835+0000","last_clean":"2024-09-18T12:09:03.055835+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:09:03.055835+0000","last_undegraded":"2024-09-18T12:09:03.055835+0000","last_fullsized":"2024-09-18T12:09:03.055835+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19463","last_scrub_stamp":"2024-09-18T12:09:03.055575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.055575+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:38:03.335605+0000","scrub_duration":0.016902436999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":93,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973654+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:09:02.973654+0000","last_peered":"2024-09-18T12:09:02.973654+0000","last_clean":"2024-09-18T12:09:02.973654+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:09:02.973654+0000","last_undegraded":"2024-09-18T12:09:02.973654+0000","last_fullsized":"2024-09-18T12:09:02.973654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26933,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978133+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:09:02.978133+0000","last_peered":"2024-09-18T12:09:02.978133+0000","last_clean":"2024-09-18T12:09:02.978133+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:09:02.978133+0000","last_undegraded":"2024-09-18T12:09:02.978133+0000","last_fullsized":"2024-09-18T12:09:02.978133+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19627,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.058553+0000","last_change":"2024-09-18T12:09:04.058553+0000","last_active":"2024-09-18T12:09:04.058553+0000","last_peered":"2024-09-18T12:09:04.058553+0000","last_clean":"2024-09-18T12:09:04.058553+0000","last_became_active":"2024-09-18T12:09:03.980052+0000","last_became_peered":"2024-09-18T12:09:03.980052+0000","last_unstale":"2024-09-18T12:09:04.058553+0000","last_undegraded":"2024-09-18T12:09:04.058553+0000","last_fullsized":"2024-09-18T12:09:04.058553+0000","mapping_epoch":60,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3538,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974525+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:09:02.974525+0000","last_peered":"2024-09-18T12:09:02.974525+0000","last_clean":"2024-09-18T12:09:02.974525+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:09:02.974525+0000","last_undegraded":"2024-09-18T12:09:02.974525+0000","last_fullsized":"2024-09-18T12:09:02.974525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26700,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.520719+0000","last_change":"2024-09-18T12:09:04.520696+0000","last_active":"2024-09-18T12:09:04.520719+0000","last_peered":"2024-09-18T12:09:04.520719+0000","last_clean":"2024-09-18T12:09:04.520719+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:09:04.520719+0000","last_undegraded":"2024-09-18T12:09:04.520719+0000","last_fullsized":"2024-09-18T12:09:04.520719+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19522","last_scrub_stamp":"2024-09-18T12:09:04.520600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:04.520600+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:36:19.536586+0000","scrub_duration":0.018151014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978460+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:09:02.978460+0000","last_peered":"2024-09-18T12:09:02.978460+0000","last_clean":"2024-09-18T12:09:02.978460+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:09:02.978460+0000","last_undegraded":"2024-09-18T12:09:02.978460+0000","last_fullsized":"2024-09-18T12:09:02.978460+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19318,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:06.520625+0000","last_change":"2024-09-18T12:09:06.520583+0000","last_active":"2024-09-18T12:09:06.520625+0000","last_peered":"2024-09-18T12:09:06.520625+0000","last_clean":"2024-09-18T12:09:06.520625+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:09:06.520625+0000","last_undegraded":"2024-09-18T12:09:06.520625+0000","last_fullsized":"2024-09-18T12:09:06.520625+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19198","last_scrub_stamp":"2024-09-18T12:09:06.520451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:06.520451+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:12:09.759949+0000","scrub_duration":0.020014351999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974547+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:09:02.974547+0000","last_peered":"2024-09-18T12:09:02.974547+0000","last_clean":"2024-09-18T12:09:02.974547+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:09:02.974547+0000","last_undegraded":"2024-09-18T12:09:02.974547+0000","last_fullsized":"2024-09-18T12:09:02.974547+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":88,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974620+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:09:02.974620+0000","last_peered":"2024-09-18T12:09:02.974620+0000","last_clean":"2024-09-18T12:09:02.974620+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:09:02.974620+0000","last_undegraded":"2024-09-18T12:09:02.974620+0000","last_fullsized":"2024-09-18T12:09:02.974620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19620,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974474+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:09:02.974474+0000","last_peered":"2024-09-18T12:09:02.974474+0000","last_clean":"2024-09-18T12:09:02.974474+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:09:02.974474+0000","last_undegraded":"2024-09-18T12:09:02.974474+0000","last_fullsized":"2024-09-18T12:09:02.974474+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6787,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217616,"num_write_kb":403276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28431,"num_bytes_recovered":7115369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90214400,"data_stored":90207442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344220,"kb_used_data":33868,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94027620,"statfs":{"total":96636764160,"available":96284282880,"internally_reserved":0,"allocated":34680832,"data_stored":34346904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.35099999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855529,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919496,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452344,"statfs":{"total":96636764160,"available":95695200256,"internally_reserved":0,"allocated":516096,"data_stored":170990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.45800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.61599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247994,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365988,"kb_used_data":55700,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94005852,"statfs":{"total":96636764160,"available":96261992448,"internally_reserved":0,"allocated":57036800,"data_stored":56691694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.52500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56033280,"data_stored":56029801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34177024,"data_stored":34177024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:08.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:clean! 2024-09-18T12:09:08.558 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-18T12:09:08.568 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-18T12:09:08.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T12:09:08.857 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:08.858 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":61,"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","created":"2024-09-18T11:53:20.028520+0000","modified":"2024-09-18T12:09:03.965797+0000","last_up_change":"2024-09-18T11:55:02.220717+0000","last_in_change":"2024-09-18T11:54:56.163654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:53:29.033564+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-09-18T11:53:40.807258+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":19,"up_from":23,"up_thru":59,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":26420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":26420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":26420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":26420}]},"public_addr":"172.21.15.154:6800/26420","cluster_addr":"172.21.15.154:6801/26420","heartbeat_back_addr":"172.21.15.154:6803/26420","heartbeat_front_addr":"172.21.15.154:6802/26420","state":["exists","up"]},{"osd":1,"uuid":"fd6ca4f9-93c5-4509-8fea-0121afa900f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":24,"up_from":35,"up_thru":60,"down_at":25,"lost_at":25,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":30839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":30839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":30839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":30839}]},"public_addr":"172.21.15.154:6808/30839","cluster_addr":"172.21.15.154:6809/30839","heartbeat_back_addr":"172.21.15.154:6811/30839","heartbeat_front_addr":"172.21.15.154:6810/30839","state":["exists","up"]},{"osd":2,"uuid":"1554e072-019d-4d8e-b2ee-c0aac5a5e811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":15,"up_from":29,"up_thru":49,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":27666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":27666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":27666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":27666}]},"public_addr":"172.21.15.154:6804/27666","cluster_addr":"172.21.15.154:6805/27666","heartbeat_back_addr":"172.21.15.154:6807/27666","heartbeat_front_addr":"172.21.15.154:6806/27666","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.685684+0000","dead_epoch":20},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.545391+0000","dead_epoch":25},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.679601+0000","dead_epoch":17}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T12:09:08.871 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T12:09:08.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:09.149 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:09.152 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:09.162 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-09-18T12:09:07.347207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6806,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217618,"num_write_kb":403278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28432,"num_bytes_recovered":7115388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1629704,"kb_used_data":90072,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281485816,"statfs":{"total":289910292480,"available":288241475584,"internally_reserved":0,"allocated":92233728,"data_stored":91209588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5553,"num_objects":-9,"num_object_clones":0,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-77579,"num_write_kb":-155141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-13163,"num_bytes_recovered":-28676096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002336"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26838,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974421+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:09:02.974421+0000","last_peered":"2024-09-18T12:09:02.974421+0000","last_clean":"2024-09-18T12:09:02.974421+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:09:02.974421+0000","last_undegraded":"2024-09-18T12:09:02.974421+0000","last_fullsized":"2024-09-18T12:09:02.974421+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19612,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978418+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:09:02.978418+0000","last_peered":"2024-09-18T12:09:02.978418+0000","last_clean":"2024-09-18T12:09:02.978418+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:09:02.978418+0000","last_undegraded":"2024-09-18T12:09:02.978418+0000","last_fullsized":"2024-09-18T12:09:02.978418+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19771,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974549+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:09:02.974549+0000","last_peered":"2024-09-18T12:09:02.974549+0000","last_clean":"2024-09-18T12:09:02.974549+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:09:02.974549+0000","last_undegraded":"2024-09-18T12:09:02.974549+0000","last_fullsized":"2024-09-18T12:09:02.974549+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19311,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978306+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:09:02.978306+0000","last_peered":"2024-09-18T12:09:02.978306+0000","last_clean":"2024-09-18T12:09:02.978306+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:09:02.978306+0000","last_undegraded":"2024-09-18T12:09:02.978306+0000","last_fullsized":"2024-09-18T12:09:02.978306+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19660,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973515+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:09:02.973515+0000","last_peered":"2024-09-18T12:09:02.973515+0000","last_clean":"2024-09-18T12:09:02.973515+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:09:02.973515+0000","last_undegraded":"2024-09-18T12:09:02.973515+0000","last_fullsized":"2024-09-18T12:09:02.973515+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19523,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974053+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:09:02.974053+0000","last_peered":"2024-09-18T12:09:02.974053+0000","last_clean":"2024-09-18T12:09:02.974053+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:09:02.974053+0000","last_undegraded":"2024-09-18T12:09:02.974053+0000","last_fullsized":"2024-09-18T12:09:02.974053+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19481,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973774+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:09:02.973774+0000","last_peered":"2024-09-18T12:09:02.973774+0000","last_clean":"2024-09-18T12:09:02.973774+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:09:02.973774+0000","last_undegraded":"2024-09-18T12:09:02.973774+0000","last_fullsized":"2024-09-18T12:09:02.973774+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973147+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:09:02.973147+0000","last_peered":"2024-09-18T12:09:02.973147+0000","last_clean":"2024-09-18T12:09:02.973147+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:09:02.973147+0000","last_undegraded":"2024-09-18T12:09:02.973147+0000","last_fullsized":"2024-09-18T12:09:02.973147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26565,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.550867+0000","last_change":"2024-09-18T12:09:03.550799+0000","last_active":"2024-09-18T12:09:03.550867+0000","last_peered":"2024-09-18T12:09:03.550867+0000","last_clean":"2024-09-18T12:09:03.550867+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:09:03.550867+0000","last_undegraded":"2024-09-18T12:09:03.550867+0000","last_fullsized":"2024-09-18T12:09:03.550867+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19405","last_scrub_stamp":"2024-09-18T12:09:03.550646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.550646+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:36:34.684302+0000","scrub_duration":0.021530328000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978072+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:09:02.978072+0000","last_peered":"2024-09-18T12:09:02.978072+0000","last_clean":"2024-09-18T12:09:02.978072+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:09:02.978072+0000","last_undegraded":"2024-09-18T12:09:02.978072+0000","last_fullsized":"2024-09-18T12:09:02.978072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26625,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973377+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:09:02.973377+0000","last_peered":"2024-09-18T12:09:02.973377+0000","last_clean":"2024-09-18T12:09:02.973377+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:09:02.973377+0000","last_undegraded":"2024-09-18T12:09:02.973377+0000","last_fullsized":"2024-09-18T12:09:02.973377+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":113,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26807,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.055835+0000","last_change":"2024-09-18T12:09:03.055754+0000","last_active":"2024-09-18T12:09:03.055835+0000","last_peered":"2024-09-18T12:09:03.055835+0000","last_clean":"2024-09-18T12:09:03.055835+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:09:03.055835+0000","last_undegraded":"2024-09-18T12:09:03.055835+0000","last_fullsized":"2024-09-18T12:09:03.055835+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19463","last_scrub_stamp":"2024-09-18T12:09:03.055575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.055575+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:38:03.335605+0000","scrub_duration":0.016902436999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":93,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973654+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:09:02.973654+0000","last_peered":"2024-09-18T12:09:02.973654+0000","last_clean":"2024-09-18T12:09:02.973654+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:09:02.973654+0000","last_undegraded":"2024-09-18T12:09:02.973654+0000","last_fullsized":"2024-09-18T12:09:02.973654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26933,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978133+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:09:02.978133+0000","last_peered":"2024-09-18T12:09:02.978133+0000","last_clean":"2024-09-18T12:09:02.978133+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:09:02.978133+0000","last_undegraded":"2024-09-18T12:09:02.978133+0000","last_fullsized":"2024-09-18T12:09:02.978133+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19627,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.058553+0000","last_change":"2024-09-18T12:09:04.058553+0000","last_active":"2024-09-18T12:09:04.058553+0000","last_peered":"2024-09-18T12:09:04.058553+0000","last_clean":"2024-09-18T12:09:04.058553+0000","last_became_active":"2024-09-18T12:09:03.980052+0000","last_became_peered":"2024-09-18T12:09:03.980052+0000","last_unstale":"2024-09-18T12:09:04.058553+0000","last_undegraded":"2024-09-18T12:09:04.058553+0000","last_fullsized":"2024-09-18T12:09:04.058553+0000","mapping_epoch":60,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3538,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974525+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:09:02.974525+0000","last_peered":"2024-09-18T12:09:02.974525+0000","last_clean":"2024-09-18T12:09:02.974525+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:09:02.974525+0000","last_undegraded":"2024-09-18T12:09:02.974525+0000","last_fullsized":"2024-09-18T12:09:02.974525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26700,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.520719+0000","last_change":"2024-09-18T12:09:04.520696+0000","last_active":"2024-09-18T12:09:04.520719+0000","last_peered":"2024-09-18T12:09:04.520719+0000","last_clean":"2024-09-18T12:09:04.520719+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:09:04.520719+0000","last_undegraded":"2024-09-18T12:09:04.520719+0000","last_fullsized":"2024-09-18T12:09:04.520719+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19522","last_scrub_stamp":"2024-09-18T12:09:04.520600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:04.520600+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:36:19.536586+0000","scrub_duration":0.018151014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978460+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:09:02.978460+0000","last_peered":"2024-09-18T12:09:02.978460+0000","last_clean":"2024-09-18T12:09:02.978460+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:09:02.978460+0000","last_undegraded":"2024-09-18T12:09:02.978460+0000","last_fullsized":"2024-09-18T12:09:02.978460+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19318,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:06.520625+0000","last_change":"2024-09-18T12:09:06.520583+0000","last_active":"2024-09-18T12:09:06.520625+0000","last_peered":"2024-09-18T12:09:06.520625+0000","last_clean":"2024-09-18T12:09:06.520625+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:09:06.520625+0000","last_undegraded":"2024-09-18T12:09:06.520625+0000","last_fullsized":"2024-09-18T12:09:06.520625+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19198","last_scrub_stamp":"2024-09-18T12:09:06.520451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:06.520451+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:12:09.759949+0000","scrub_duration":0.020014351999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974547+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:09:02.974547+0000","last_peered":"2024-09-18T12:09:02.974547+0000","last_clean":"2024-09-18T12:09:02.974547+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:09:02.974547+0000","last_undegraded":"2024-09-18T12:09:02.974547+0000","last_fullsized":"2024-09-18T12:09:02.974547+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":88,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974620+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:09:02.974620+0000","last_peered":"2024-09-18T12:09:02.974620+0000","last_clean":"2024-09-18T12:09:02.974620+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:09:02.974620+0000","last_undegraded":"2024-09-18T12:09:02.974620+0000","last_fullsized":"2024-09-18T12:09:02.974620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19620,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974474+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:09:02.974474+0000","last_peered":"2024-09-18T12:09:02.974474+0000","last_clean":"2024-09-18T12:09:02.974474+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:09:02.974474+0000","last_undegraded":"2024-09-18T12:09:02.974474+0000","last_fullsized":"2024-09-18T12:09:02.974474+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6787,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217616,"num_write_kb":403276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28431,"num_bytes_recovered":7115369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90214400,"data_stored":90207442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344220,"kb_used_data":33868,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94027620,"statfs":{"total":96636764160,"available":96284282880,"internally_reserved":0,"allocated":34680832,"data_stored":34346904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.35099999999999998}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.46300000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855529,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919496,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452344,"statfs":{"total":96636764160,"available":95695200256,"internally_reserved":0,"allocated":516096,"data_stored":170990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.45800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.61599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247994,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365988,"kb_used_data":55700,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94005852,"statfs":{"total":96636764160,"available":96261992448,"internally_reserved":0,"allocated":57036800,"data_stored":56691694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.52500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56033280,"data_stored":56029801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34177024,"data_stored":34177024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:09.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:09.445 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:09.448 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:09.458 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2024-09-18T12:09:09.347795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6806,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217618,"num_write_kb":403278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28432,"num_bytes_recovered":7115388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1627416,"kb_used_data":87792,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281488104,"statfs":{"total":289910292480,"available":288243818496,"internally_reserved":0,"allocated":89899008,"data_stored":88874868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5553,"num_objects":-9,"num_object_clones":0,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-77579,"num_write_kb":-155141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-13992,"num_bytes_recovered":-28676096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002312"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26838,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974421+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:09:02.974421+0000","last_peered":"2024-09-18T12:09:02.974421+0000","last_clean":"2024-09-18T12:09:02.974421+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:09:02.974421+0000","last_undegraded":"2024-09-18T12:09:02.974421+0000","last_fullsized":"2024-09-18T12:09:02.974421+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19612,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978418+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:09:02.978418+0000","last_peered":"2024-09-18T12:09:02.978418+0000","last_clean":"2024-09-18T12:09:02.978418+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:09:02.978418+0000","last_undegraded":"2024-09-18T12:09:02.978418+0000","last_fullsized":"2024-09-18T12:09:02.978418+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19771,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974549+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:09:02.974549+0000","last_peered":"2024-09-18T12:09:02.974549+0000","last_clean":"2024-09-18T12:09:02.974549+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:09:02.974549+0000","last_undegraded":"2024-09-18T12:09:02.974549+0000","last_fullsized":"2024-09-18T12:09:02.974549+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19311,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978306+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:09:02.978306+0000","last_peered":"2024-09-18T12:09:02.978306+0000","last_clean":"2024-09-18T12:09:02.978306+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:09:02.978306+0000","last_undegraded":"2024-09-18T12:09:02.978306+0000","last_fullsized":"2024-09-18T12:09:02.978306+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19660,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973515+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:09:02.973515+0000","last_peered":"2024-09-18T12:09:02.973515+0000","last_clean":"2024-09-18T12:09:02.973515+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:09:02.973515+0000","last_undegraded":"2024-09-18T12:09:02.973515+0000","last_fullsized":"2024-09-18T12:09:02.973515+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19523,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974053+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:09:02.974053+0000","last_peered":"2024-09-18T12:09:02.974053+0000","last_clean":"2024-09-18T12:09:02.974053+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:09:02.974053+0000","last_undegraded":"2024-09-18T12:09:02.974053+0000","last_fullsized":"2024-09-18T12:09:02.974053+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19481,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973774+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:09:02.973774+0000","last_peered":"2024-09-18T12:09:02.973774+0000","last_clean":"2024-09-18T12:09:02.973774+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:09:02.973774+0000","last_undegraded":"2024-09-18T12:09:02.973774+0000","last_fullsized":"2024-09-18T12:09:02.973774+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973147+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:09:02.973147+0000","last_peered":"2024-09-18T12:09:02.973147+0000","last_clean":"2024-09-18T12:09:02.973147+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:09:02.973147+0000","last_undegraded":"2024-09-18T12:09:02.973147+0000","last_fullsized":"2024-09-18T12:09:02.973147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26565,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.550867+0000","last_change":"2024-09-18T12:09:03.550799+0000","last_active":"2024-09-18T12:09:03.550867+0000","last_peered":"2024-09-18T12:09:03.550867+0000","last_clean":"2024-09-18T12:09:03.550867+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:09:03.550867+0000","last_undegraded":"2024-09-18T12:09:03.550867+0000","last_fullsized":"2024-09-18T12:09:03.550867+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19405","last_scrub_stamp":"2024-09-18T12:09:03.550646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.550646+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:36:34.684302+0000","scrub_duration":0.021530328000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978072+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:09:02.978072+0000","last_peered":"2024-09-18T12:09:02.978072+0000","last_clean":"2024-09-18T12:09:02.978072+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:09:02.978072+0000","last_undegraded":"2024-09-18T12:09:02.978072+0000","last_fullsized":"2024-09-18T12:09:02.978072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26625,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973377+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:09:02.973377+0000","last_peered":"2024-09-18T12:09:02.973377+0000","last_clean":"2024-09-18T12:09:02.973377+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:09:02.973377+0000","last_undegraded":"2024-09-18T12:09:02.973377+0000","last_fullsized":"2024-09-18T12:09:02.973377+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":113,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26807,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.055835+0000","last_change":"2024-09-18T12:09:03.055754+0000","last_active":"2024-09-18T12:09:03.055835+0000","last_peered":"2024-09-18T12:09:03.055835+0000","last_clean":"2024-09-18T12:09:03.055835+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:09:03.055835+0000","last_undegraded":"2024-09-18T12:09:03.055835+0000","last_fullsized":"2024-09-18T12:09:03.055835+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19463","last_scrub_stamp":"2024-09-18T12:09:03.055575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.055575+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:38:03.335605+0000","scrub_duration":0.016902436999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":93,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973654+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:09:02.973654+0000","last_peered":"2024-09-18T12:09:02.973654+0000","last_clean":"2024-09-18T12:09:02.973654+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:09:02.973654+0000","last_undegraded":"2024-09-18T12:09:02.973654+0000","last_fullsized":"2024-09-18T12:09:02.973654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26933,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978133+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:09:02.978133+0000","last_peered":"2024-09-18T12:09:02.978133+0000","last_clean":"2024-09-18T12:09:02.978133+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:09:02.978133+0000","last_undegraded":"2024-09-18T12:09:02.978133+0000","last_fullsized":"2024-09-18T12:09:02.978133+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19627,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.058553+0000","last_change":"2024-09-18T12:09:04.058553+0000","last_active":"2024-09-18T12:09:04.058553+0000","last_peered":"2024-09-18T12:09:04.058553+0000","last_clean":"2024-09-18T12:09:04.058553+0000","last_became_active":"2024-09-18T12:09:03.980052+0000","last_became_peered":"2024-09-18T12:09:03.980052+0000","last_unstale":"2024-09-18T12:09:04.058553+0000","last_undegraded":"2024-09-18T12:09:04.058553+0000","last_fullsized":"2024-09-18T12:09:04.058553+0000","mapping_epoch":60,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3538,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974525+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:09:02.974525+0000","last_peered":"2024-09-18T12:09:02.974525+0000","last_clean":"2024-09-18T12:09:02.974525+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:09:02.974525+0000","last_undegraded":"2024-09-18T12:09:02.974525+0000","last_fullsized":"2024-09-18T12:09:02.974525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26700,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.520719+0000","last_change":"2024-09-18T12:09:04.520696+0000","last_active":"2024-09-18T12:09:04.520719+0000","last_peered":"2024-09-18T12:09:04.520719+0000","last_clean":"2024-09-18T12:09:04.520719+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:09:04.520719+0000","last_undegraded":"2024-09-18T12:09:04.520719+0000","last_fullsized":"2024-09-18T12:09:04.520719+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19522","last_scrub_stamp":"2024-09-18T12:09:04.520600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:04.520600+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:36:19.536586+0000","scrub_duration":0.018151014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978460+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:09:02.978460+0000","last_peered":"2024-09-18T12:09:02.978460+0000","last_clean":"2024-09-18T12:09:02.978460+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:09:02.978460+0000","last_undegraded":"2024-09-18T12:09:02.978460+0000","last_fullsized":"2024-09-18T12:09:02.978460+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19318,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:06.520625+0000","last_change":"2024-09-18T12:09:06.520583+0000","last_active":"2024-09-18T12:09:06.520625+0000","last_peered":"2024-09-18T12:09:06.520625+0000","last_clean":"2024-09-18T12:09:06.520625+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:09:06.520625+0000","last_undegraded":"2024-09-18T12:09:06.520625+0000","last_fullsized":"2024-09-18T12:09:06.520625+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19198","last_scrub_stamp":"2024-09-18T12:09:06.520451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:06.520451+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:12:09.759949+0000","scrub_duration":0.020014351999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974547+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:09:02.974547+0000","last_peered":"2024-09-18T12:09:02.974547+0000","last_clean":"2024-09-18T12:09:02.974547+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:09:02.974547+0000","last_undegraded":"2024-09-18T12:09:02.974547+0000","last_fullsized":"2024-09-18T12:09:02.974547+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":88,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974620+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:09:02.974620+0000","last_peered":"2024-09-18T12:09:02.974620+0000","last_clean":"2024-09-18T12:09:02.974620+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:09:02.974620+0000","last_undegraded":"2024-09-18T12:09:02.974620+0000","last_fullsized":"2024-09-18T12:09:02.974620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19620,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974474+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:09:02.974474+0000","last_peered":"2024-09-18T12:09:02.974474+0000","last_clean":"2024-09-18T12:09:02.974474+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:09:02.974474+0000","last_undegraded":"2024-09-18T12:09:02.974474+0000","last_fullsized":"2024-09-18T12:09:02.974474+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6787,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217616,"num_write_kb":403276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28431,"num_bytes_recovered":7115369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87756800,"data_stored":87749842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341932,"kb_used_data":31588,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94029908,"statfs":{"total":96636764160,"available":96286625792,"internally_reserved":0,"allocated":32346112,"data_stored":32012184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.497},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.441}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.39900000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855529,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919496,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452344,"statfs":{"total":96636764160,"available":95695200256,"internally_reserved":0,"allocated":516096,"data_stored":170990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.45800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.61599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247994,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365988,"kb_used_data":55700,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94005852,"statfs":{"total":96636764160,"available":96261992448,"internally_reserved":0,"allocated":57036800,"data_stored":56691694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.52500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56033280,"data_stored":56029801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":31719424,"data_stored":31719424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:09.459 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T12:09:09.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:09.740 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:09.751 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2024-09-18T12:09:09.347795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6806,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217618,"num_write_kb":403278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28432,"num_bytes_recovered":7115388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1627416,"kb_used_data":87792,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281488104,"statfs":{"total":289910292480,"available":288243818496,"internally_reserved":0,"allocated":89899008,"data_stored":88874868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5553,"num_objects":-9,"num_object_clones":0,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-77579,"num_write_kb":-155141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-13992,"num_bytes_recovered":-28676096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002312"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26838,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974421+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:09:02.974421+0000","last_peered":"2024-09-18T12:09:02.974421+0000","last_clean":"2024-09-18T12:09:02.974421+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:09:02.974421+0000","last_undegraded":"2024-09-18T12:09:02.974421+0000","last_fullsized":"2024-09-18T12:09:02.974421+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19612,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978418+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:09:02.978418+0000","last_peered":"2024-09-18T12:09:02.978418+0000","last_clean":"2024-09-18T12:09:02.978418+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:09:02.978418+0000","last_undegraded":"2024-09-18T12:09:02.978418+0000","last_fullsized":"2024-09-18T12:09:02.978418+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19771,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974549+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:09:02.974549+0000","last_peered":"2024-09-18T12:09:02.974549+0000","last_clean":"2024-09-18T12:09:02.974549+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:09:02.974549+0000","last_undegraded":"2024-09-18T12:09:02.974549+0000","last_fullsized":"2024-09-18T12:09:02.974549+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19311,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978306+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:09:02.978306+0000","last_peered":"2024-09-18T12:09:02.978306+0000","last_clean":"2024-09-18T12:09:02.978306+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:09:02.978306+0000","last_undegraded":"2024-09-18T12:09:02.978306+0000","last_fullsized":"2024-09-18T12:09:02.978306+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19660,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973515+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:09:02.973515+0000","last_peered":"2024-09-18T12:09:02.973515+0000","last_clean":"2024-09-18T12:09:02.973515+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:09:02.973515+0000","last_undegraded":"2024-09-18T12:09:02.973515+0000","last_fullsized":"2024-09-18T12:09:02.973515+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19523,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974053+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:09:02.974053+0000","last_peered":"2024-09-18T12:09:02.974053+0000","last_clean":"2024-09-18T12:09:02.974053+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:09:02.974053+0000","last_undegraded":"2024-09-18T12:09:02.974053+0000","last_fullsized":"2024-09-18T12:09:02.974053+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19481,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973774+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:09:02.973774+0000","last_peered":"2024-09-18T12:09:02.973774+0000","last_clean":"2024-09-18T12:09:02.973774+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:09:02.973774+0000","last_undegraded":"2024-09-18T12:09:02.973774+0000","last_fullsized":"2024-09-18T12:09:02.973774+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973147+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:09:02.973147+0000","last_peered":"2024-09-18T12:09:02.973147+0000","last_clean":"2024-09-18T12:09:02.973147+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:09:02.973147+0000","last_undegraded":"2024-09-18T12:09:02.973147+0000","last_fullsized":"2024-09-18T12:09:02.973147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26565,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.550867+0000","last_change":"2024-09-18T12:09:03.550799+0000","last_active":"2024-09-18T12:09:03.550867+0000","last_peered":"2024-09-18T12:09:03.550867+0000","last_clean":"2024-09-18T12:09:03.550867+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:09:03.550867+0000","last_undegraded":"2024-09-18T12:09:03.550867+0000","last_fullsized":"2024-09-18T12:09:03.550867+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19405","last_scrub_stamp":"2024-09-18T12:09:03.550646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.550646+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:36:34.684302+0000","scrub_duration":0.021530328000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978072+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:09:02.978072+0000","last_peered":"2024-09-18T12:09:02.978072+0000","last_clean":"2024-09-18T12:09:02.978072+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:09:02.978072+0000","last_undegraded":"2024-09-18T12:09:02.978072+0000","last_fullsized":"2024-09-18T12:09:02.978072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26625,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973377+0000","last_change":"2024-09-18T12:04:39.218592+0000","last_active":"2024-09-18T12:09:02.973377+0000","last_peered":"2024-09-18T12:09:02.973377+0000","last_clean":"2024-09-18T12:09:02.973377+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:09:02.973377+0000","last_undegraded":"2024-09-18T12:09:02.973377+0000","last_fullsized":"2024-09-18T12:09:02.973377+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19444,"num_write_kb":38880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":113,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26807,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.055835+0000","last_change":"2024-09-18T12:09:03.055754+0000","last_active":"2024-09-18T12:09:03.055835+0000","last_peered":"2024-09-18T12:09:03.055835+0000","last_clean":"2024-09-18T12:09:03.055835+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:09:03.055835+0000","last_undegraded":"2024-09-18T12:09:03.055835+0000","last_fullsized":"2024-09-18T12:09:03.055835+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19463","last_scrub_stamp":"2024-09-18T12:09:03.055575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.055575+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:38:03.335605+0000","scrub_duration":0.016902436999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":93,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973654+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:09:02.973654+0000","last_peered":"2024-09-18T12:09:02.973654+0000","last_clean":"2024-09-18T12:09:02.973654+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:09:02.973654+0000","last_undegraded":"2024-09-18T12:09:02.973654+0000","last_fullsized":"2024-09-18T12:09:02.973654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26933,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978133+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:09:02.978133+0000","last_peered":"2024-09-18T12:09:02.978133+0000","last_clean":"2024-09-18T12:09:02.978133+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:09:02.978133+0000","last_undegraded":"2024-09-18T12:09:02.978133+0000","last_fullsized":"2024-09-18T12:09:02.978133+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19627,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.058553+0000","last_change":"2024-09-18T12:09:04.058553+0000","last_active":"2024-09-18T12:09:04.058553+0000","last_peered":"2024-09-18T12:09:04.058553+0000","last_clean":"2024-09-18T12:09:04.058553+0000","last_became_active":"2024-09-18T12:09:03.980052+0000","last_became_peered":"2024-09-18T12:09:03.980052+0000","last_unstale":"2024-09-18T12:09:04.058553+0000","last_undegraded":"2024-09-18T12:09:04.058553+0000","last_fullsized":"2024-09-18T12:09:04.058553+0000","mapping_epoch":60,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3538,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974525+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:09:02.974525+0000","last_peered":"2024-09-18T12:09:02.974525+0000","last_clean":"2024-09-18T12:09:02.974525+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:09:02.974525+0000","last_undegraded":"2024-09-18T12:09:02.974525+0000","last_fullsized":"2024-09-18T12:09:02.974525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26700,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.520719+0000","last_change":"2024-09-18T12:09:04.520696+0000","last_active":"2024-09-18T12:09:04.520719+0000","last_peered":"2024-09-18T12:09:04.520719+0000","last_clean":"2024-09-18T12:09:04.520719+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:09:04.520719+0000","last_undegraded":"2024-09-18T12:09:04.520719+0000","last_fullsized":"2024-09-18T12:09:04.520719+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19522","last_scrub_stamp":"2024-09-18T12:09:04.520600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:04.520600+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:36:19.536586+0000","scrub_duration":0.018151014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978460+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:09:02.978460+0000","last_peered":"2024-09-18T12:09:02.978460+0000","last_clean":"2024-09-18T12:09:02.978460+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:09:02.978460+0000","last_undegraded":"2024-09-18T12:09:02.978460+0000","last_fullsized":"2024-09-18T12:09:02.978460+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19318,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:06.520625+0000","last_change":"2024-09-18T12:09:06.520583+0000","last_active":"2024-09-18T12:09:06.520625+0000","last_peered":"2024-09-18T12:09:06.520625+0000","last_clean":"2024-09-18T12:09:06.520625+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:09:06.520625+0000","last_undegraded":"2024-09-18T12:09:06.520625+0000","last_fullsized":"2024-09-18T12:09:06.520625+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19198","last_scrub_stamp":"2024-09-18T12:09:06.520451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:06.520451+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:12:09.759949+0000","scrub_duration":0.020014351999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974547+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:09:02.974547+0000","last_peered":"2024-09-18T12:09:02.974547+0000","last_clean":"2024-09-18T12:09:02.974547+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:09:02.974547+0000","last_undegraded":"2024-09-18T12:09:02.974547+0000","last_fullsized":"2024-09-18T12:09:02.974547+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":88,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974620+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:09:02.974620+0000","last_peered":"2024-09-18T12:09:02.974620+0000","last_clean":"2024-09-18T12:09:02.974620+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:09:02.974620+0000","last_undegraded":"2024-09-18T12:09:02.974620+0000","last_fullsized":"2024-09-18T12:09:02.974620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19620,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974474+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:09:02.974474+0000","last_peered":"2024-09-18T12:09:02.974474+0000","last_clean":"2024-09-18T12:09:02.974474+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:09:02.974474+0000","last_undegraded":"2024-09-18T12:09:02.974474+0000","last_fullsized":"2024-09-18T12:09:02.974474+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6787,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217616,"num_write_kb":403276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28431,"num_bytes_recovered":7115369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87756800,"data_stored":87749842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341932,"kb_used_data":31588,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94029908,"statfs":{"total":96636764160,"available":96286625792,"internally_reserved":0,"allocated":32346112,"data_stored":32012184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.497},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.441}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.39900000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855529,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919496,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452344,"statfs":{"total":96636764160,"available":95695200256,"internally_reserved":0,"allocated":516096,"data_stored":170990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.45800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.61599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247994,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":365988,"kb_used_data":55700,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94005852,"statfs":{"total":96636764160,"available":96261992448,"internally_reserved":0,"allocated":57036800,"data_stored":56691694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.52500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56033280,"data_stored":56029801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":31719424,"data_stored":31719424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:09.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T12:09:10.036 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:10.036 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":61,"fsid":"c6912087-b15b-4223-a3df-3a06756ee775","created":"2024-09-18T11:53:20.028520+0000","modified":"2024-09-18T12:09:03.965797+0000","last_up_change":"2024-09-18T11:55:02.220717+0000","last_in_change":"2024-09-18T11:54:56.163654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T11:53:29.033564+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-09-18T11:53:40.807258+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"db9cbbc8-04f4-44e6-b26b-e0a9e1a1391f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":19,"up_from":23,"up_thru":59,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":26420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":26420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":26420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":26420}]},"public_addr":"172.21.15.154:6800/26420","cluster_addr":"172.21.15.154:6801/26420","heartbeat_back_addr":"172.21.15.154:6803/26420","heartbeat_front_addr":"172.21.15.154:6802/26420","state":["exists","up"]},{"osd":1,"uuid":"fd6ca4f9-93c5-4509-8fea-0121afa900f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":24,"up_from":35,"up_thru":60,"down_at":25,"lost_at":25,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":30839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":30839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":30839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":30839}]},"public_addr":"172.21.15.154:6808/30839","cluster_addr":"172.21.15.154:6809/30839","heartbeat_back_addr":"172.21.15.154:6811/30839","heartbeat_front_addr":"172.21.15.154:6810/30839","state":["exists","up"]},{"osd":2,"uuid":"1554e072-019d-4d8e-b2ee-c0aac5a5e811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":15,"up_from":29,"up_thru":49,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":27666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":27666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":27666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":27666}]},"public_addr":"172.21.15.154:6804/27666","cluster_addr":"172.21.15.154:6805/27666","heartbeat_back_addr":"172.21.15.154:6807/27666","heartbeat_front_addr":"172.21.15.154:6806/27666","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.685684+0000","dead_epoch":20},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.545391+0000","dead_epoch":25},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T11:53:26.679601+0000","dead_epoch":17}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T12:09:11.050 INFO:tasks.ceph:Scrubbing osd.0 2024-09-18T12:09:11.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:09:11.200 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T12:09:11.200 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:09:11.200 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T12:09:11.213 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-18T12:09:11.507 INFO:teuthology.orchestra.run.smithi154.stderr:instructed osd(s) 0 to deep-scrub 2024-09-18T12:09:11.521 INFO:tasks.ceph:Scrubbing osd.1 2024-09-18T12:09:11.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:09:11.653 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T12:09:11.653 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:09:11.653 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T12:09:11.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-18T12:09:11.959 INFO:teuthology.orchestra.run.smithi154.stderr:instructed osd(s) 1 to deep-scrub 2024-09-18T12:09:11.973 INFO:tasks.ceph:Scrubbing osd.2 2024-09-18T12:09:11.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-18T12:09:12.113 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-18T12:09:12.113 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T12:09:12.113 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-18T12:09:12.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-18T12:09:12.418 INFO:teuthology.orchestra.run.smithi154.stderr:instructed osd(s) 2 to deep-scrub 2024-09-18T12:09:12.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:12.716 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:12.718 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:12.729 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":530,"stamp":"2024-09-18T12:09:11.348253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5572,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198174,"num_write_kb":364398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24896,"num_bytes_recovered":7115388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160705,"ondisk_log_size":160705,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1624528,"kb_used_data":84912,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281490992,"statfs":{"total":289910292480,"available":288246775808,"internally_reserved":0,"allocated":86949888,"data_stored":85925748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6787,"num_objects":-11,"num_object_clones":0,"num_object_copies":-22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-97023,"num_write_kb":-194021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-17528,"num_bytes_recovered":-28676096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.379698"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26838,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974421+0000","last_change":"2024-09-18T12:07:56.734610+0000","last_active":"2024-09-18T12:09:02.974421+0000","last_peered":"2024-09-18T12:09:02.974421+0000","last_clean":"2024-09-18T12:09:02.974421+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:09:02.974421+0000","last_undegraded":"2024-09-18T12:09:02.974421+0000","last_fullsized":"2024-09-18T12:09:02.974421+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:14:18.784499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19612,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978418+0000","last_change":"2024-09-18T11:54:51.317255+0000","last_active":"2024-09-18T12:09:02.978418+0000","last_peered":"2024-09-18T12:09:02.978418+0000","last_clean":"2024-09-18T12:09:02.978418+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:09:02.978418+0000","last_undegraded":"2024-09-18T12:09:02.978418+0000","last_fullsized":"2024-09-18T12:09:02.978418+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2893","last_scrub_stamp":"2024-09-18T11:54:51.317158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.317158+0000","objects_scrubbed":2802,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:38:53.860220+0000","scrub_duration":1.731777508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19771,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974549+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:09:02.974549+0000","last_peered":"2024-09-18T12:09:02.974549+0000","last_clean":"2024-09-18T12:09:02.974549+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:09:02.974549+0000","last_undegraded":"2024-09-18T12:09:02.974549+0000","last_fullsized":"2024-09-18T12:09:02.974549+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:47:34.496388+0000","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19311,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978306+0000","last_change":"2024-09-18T11:54:18.439837+0000","last_active":"2024-09-18T12:09:02.978306+0000","last_peered":"2024-09-18T12:09:02.978306+0000","last_clean":"2024-09-18T12:09:02.978306+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:09:02.978306+0000","last_undegraded":"2024-09-18T12:09:02.978306+0000","last_fullsized":"2024-09-18T12:09:02.978306+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:44:22.218985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19660,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973515+0000","last_change":"2024-09-18T11:54:51.789889+0000","last_active":"2024-09-18T12:09:02.973515+0000","last_peered":"2024-09-18T12:09:02.973515+0000","last_clean":"2024-09-18T12:09:02.973515+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:09:02.973515+0000","last_undegraded":"2024-09-18T12:09:02.973515+0000","last_fullsized":"2024-09-18T12:09:02.973515+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2895","last_scrub_stamp":"2024-09-18T11:54:51.789742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:51.789742+0000","objects_scrubbed":2828,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:17:00.658217+0000","scrub_duration":1.407836581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19523,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974053+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:09:02.974053+0000","last_peered":"2024-09-18T12:09:02.974053+0000","last_clean":"2024-09-18T12:09:02.974053+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:09:02.974053+0000","last_undegraded":"2024-09-18T12:09:02.974053+0000","last_fullsized":"2024-09-18T12:09:02.974053+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19481,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973774+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:09:02.973774+0000","last_peered":"2024-09-18T12:09:02.973774+0000","last_clean":"2024-09-18T12:09:02.973774+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:09:02.973774+0000","last_undegraded":"2024-09-18T12:09:02.973774+0000","last_fullsized":"2024-09-18T12:09:02.973774+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19849,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:03.848368+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3683","last_scrub_stamp":"2024-09-18T11:55:03.848218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:03.848218+0000","objects_scrubbed":3681,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:03:41.897487+0000","scrub_duration":0.58095081199999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973147+0000","last_change":"2024-09-18T11:55:04.264218+0000","last_active":"2024-09-18T12:09:02.973147+0000","last_peered":"2024-09-18T12:09:02.973147+0000","last_clean":"2024-09-18T12:09:02.973147+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:09:02.973147+0000","last_undegraded":"2024-09-18T12:09:02.973147+0000","last_fullsized":"2024-09-18T12:09:02.973147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:21:13.249954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26565,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.550867+0000","last_change":"2024-09-18T12:09:03.550799+0000","last_active":"2024-09-18T12:09:03.550867+0000","last_peered":"2024-09-18T12:09:03.550867+0000","last_clean":"2024-09-18T12:09:03.550867+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:09:03.550867+0000","last_undegraded":"2024-09-18T12:09:03.550867+0000","last_fullsized":"2024-09-18T12:09:03.550867+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19405","last_scrub_stamp":"2024-09-18T12:09:03.550646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.550646+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:36:34.684302+0000","scrub_duration":0.021530328000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978072+0000","last_change":"2024-09-18T11:54:18.439949+0000","last_active":"2024-09-18T12:09:02.978072+0000","last_peered":"2024-09-18T12:09:02.978072+0000","last_clean":"2024-09-18T12:09:02.978072+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:09:02.978072+0000","last_undegraded":"2024-09-18T12:09:02.978072+0000","last_fullsized":"2024-09-18T12:09:02.978072+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:38:40.148425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26635,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:10.711437+0000","last_change":"2024-09-18T12:09:10.711408+0000","last_active":"2024-09-18T12:09:10.711437+0000","last_peered":"2024-09-18T12:09:10.711437+0000","last_clean":"2024-09-18T12:09:10.711437+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:09:10.711437+0000","last_undegraded":"2024-09-18T12:09:10.711437+0000","last_fullsized":"2024-09-18T12:09:10.711437+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19446","last_scrub_stamp":"2024-09-18T12:09:10.711316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:10.711316+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:07:57.981413+0000","scrub_duration":0.019690760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":113,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973280+0000","last_change":"2024-09-18T11:55:04.715438+0000","last_active":"2024-09-18T12:09:02.973280+0000","last_peered":"2024-09-18T12:09:02.973280+0000","last_clean":"2024-09-18T12:09:02.973280+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:09:02.973280+0000","last_undegraded":"2024-09-18T12:09:02.973280+0000","last_fullsized":"2024-09-18T12:09:02.973280+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:25:02.656554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26807,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.055835+0000","last_change":"2024-09-18T12:09:03.055754+0000","last_active":"2024-09-18T12:09:03.055835+0000","last_peered":"2024-09-18T12:09:03.055835+0000","last_clean":"2024-09-18T12:09:03.055835+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:09:03.055835+0000","last_undegraded":"2024-09-18T12:09:03.055835+0000","last_fullsized":"2024-09-18T12:09:03.055835+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19463","last_scrub_stamp":"2024-09-18T12:09:03.055575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.055575+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:38:03.335605+0000","scrub_duration":0.016902436999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":93,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.973654+0000","last_change":"2024-09-18T11:55:04.343861+0000","last_active":"2024-09-18T12:09:02.973654+0000","last_peered":"2024-09-18T12:09:02.973654+0000","last_clean":"2024-09-18T12:09:02.973654+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:09:02.973654+0000","last_undegraded":"2024-09-18T12:09:02.973654+0000","last_fullsized":"2024-09-18T12:09:02.973654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:29:16.182429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26933,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978133+0000","last_change":"2024-09-18T12:06:54.864342+0000","last_active":"2024-09-18T12:09:02.978133+0000","last_peered":"2024-09-18T12:09:02.978133+0000","last_clean":"2024-09-18T12:09:02.978133+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:09:02.978133+0000","last_undegraded":"2024-09-18T12:09:02.978133+0000","last_fullsized":"2024-09-18T12:09:02.978133+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3593","last_scrub_stamp":"2024-09-18T11:55:02.143655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:55:02.143655+0000","objects_scrubbed":3514,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:37:02.037554+0000","scrub_duration":2.374930532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19627,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.058553+0000","last_change":"2024-09-18T12:09:04.058553+0000","last_active":"2024-09-18T12:09:04.058553+0000","last_peered":"2024-09-18T12:09:04.058553+0000","last_clean":"2024-09-18T12:09:04.058553+0000","last_became_active":"2024-09-18T12:09:03.980052+0000","last_became_peered":"2024-09-18T12:09:03.980052+0000","last_unstale":"2024-09-18T12:09:04.058553+0000","last_undegraded":"2024-09-18T12:09:04.058553+0000","last_fullsized":"2024-09-18T12:09:04.058553+0000","mapping_epoch":60,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3538,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974525+0000","last_change":"2024-09-18T11:55:04.451721+0000","last_active":"2024-09-18T12:09:02.974525+0000","last_peered":"2024-09-18T12:09:02.974525+0000","last_clean":"2024-09-18T12:09:02.974525+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:09:02.974525+0000","last_undegraded":"2024-09-18T12:09:02.974525+0000","last_fullsized":"2024-09-18T12:09:02.974525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:45:32.805495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26700,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.520719+0000","last_change":"2024-09-18T12:09:04.520696+0000","last_active":"2024-09-18T12:09:04.520719+0000","last_peered":"2024-09-18T12:09:04.520719+0000","last_clean":"2024-09-18T12:09:04.520719+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:09:04.520719+0000","last_undegraded":"2024-09-18T12:09:04.520719+0000","last_fullsized":"2024-09-18T12:09:04.520719+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19522","last_scrub_stamp":"2024-09-18T12:09:04.520600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:04.520600+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:36:19.536586+0000","scrub_duration":0.018151014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.978460+0000","last_change":"2024-09-18T11:54:18.440580+0000","last_active":"2024-09-18T12:09:02.978460+0000","last_peered":"2024-09-18T12:09:02.978460+0000","last_clean":"2024-09-18T12:09:02.978460+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:09:02.978460+0000","last_undegraded":"2024-09-18T12:09:02.978460+0000","last_fullsized":"2024-09-18T12:09:02.978460+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:26:58.912835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19318,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:06.520625+0000","last_change":"2024-09-18T12:09:06.520583+0000","last_active":"2024-09-18T12:09:06.520625+0000","last_peered":"2024-09-18T12:09:06.520625+0000","last_clean":"2024-09-18T12:09:06.520625+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:09:06.520625+0000","last_undegraded":"2024-09-18T12:09:06.520625+0000","last_fullsized":"2024-09-18T12:09:06.520625+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19198","last_scrub_stamp":"2024-09-18T12:09:06.520451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:06.520451+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:12:09.759949+0000","scrub_duration":0.020014351999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974547+0000","last_change":"2024-09-18T11:55:04.427803+0000","last_active":"2024-09-18T12:09:02.974547+0000","last_peered":"2024-09-18T12:09:02.974547+0000","last_clean":"2024-09-18T12:09:02.974547+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:09:02.974547+0000","last_undegraded":"2024-09-18T12:09:02.974547+0000","last_fullsized":"2024-09-18T12:09:02.974547+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:50:24.144255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":88,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974620+0000","last_change":"2024-09-18T11:55:04.256023+0000","last_active":"2024-09-18T12:09:02.974620+0000","last_peered":"2024-09-18T12:09:02.974620+0000","last_clean":"2024-09-18T12:09:02.974620+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:09:02.974620+0000","last_undegraded":"2024-09-18T12:09:02.974620+0000","last_fullsized":"2024-09-18T12:09:02.974620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:29.091408+0000","last_clean_scrub_stamp":"2024-09-18T11:53:29.091408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:27:41.053841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19620,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-18T12:09:02.974474+0000","last_change":"2024-09-18T12:06:56.207646+0000","last_active":"2024-09-18T12:09:02.974474+0000","last_peered":"2024-09-18T12:09:02.974474+0000","last_clean":"2024-09-18T12:09:02.974474+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:09:02.974474+0000","last_undegraded":"2024-09-18T12:09:02.974474+0000","last_fullsized":"2024-09-18T12:09:02.974474+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2468,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19509,"num_write_kb":39007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3530,"num_bytes_recovered":617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5553,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198172,"num_write_kb":364396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24895,"num_bytes_recovered":7115369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":85299200,"data_stored":85292242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341932,"kb_used_data":31588,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94029908,"statfs":{"total":96636764160,"available":96286625792,"internally_reserved":0,"allocated":32346112,"data_stored":32012184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.497},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.441}]},{"osd":1,"last update":"Wed Sep 18 12:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.94899999999999995,"15min":1.4490000000000001},"min":{"1min":0.33800000000000002,"5min":0.221,"15min":0.20999999999999999},"max":{"1min":0.58799999999999997,"5min":16.847999999999999,"15min":16.847999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.76400000000000001,"15min":1.0349999999999999},"min":{"1min":0.34999999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63500000000000001,"5min":8.0449999999999999,"15min":12.712},"last":0.39900000000000002}]}]},{"osd":1,"up_from":35,"seq":150323855529,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919496,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452344,"statfs":{"total":96636764160,"available":95695200256,"internally_reserved":0,"allocated":516096,"data_stored":170990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.45800000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.61599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247995,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363100,"kb_used_data":52820,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94008740,"statfs":{"total":96636764160,"available":96264949760,"internally_reserved":0,"allocated":54087680,"data_stored":53742574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.60699999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.57399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53575680,"data_stored":53572201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":31719424,"data_stored":31719424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:12.734 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-09-18T11:53:41.166841+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=41, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.734 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-09-18T11:54:51.317158+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=54, tm_sec=51, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.734 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-09-18T11:54:55.665140+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=54, tm_sec=55, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.734 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-09-18T11:53:41.166841+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=41, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.734 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-09-18T11:54:51.789742+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=54, tm_sec=51, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.734 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-09-18T11:53:41.166841+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=41, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.734 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-09-18T11:53:41.166841+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=41, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.734 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-09-18T11:55:03.848218+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=55, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.735 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-18T11:53:29.091408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=29, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.735 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-18T12:09:03.550646+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.735 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-18T11:53:29.091408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=29, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.735 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-18T12:09:10.711316+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.735 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-18T11:53:29.091408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=29, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.735 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-18T12:09:03.055575+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.735 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-18T11:53:29.091408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=29, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.735 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-18T11:55:02.143655+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=55, tm_sec=2, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.735 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-18T11:53:41.166841+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=41, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.736 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-18T11:53:29.091408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=29, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.736 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-18T12:09:04.520600+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=4, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.736 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-18T11:53:29.091408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=29, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.736 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-18T12:09:06.520451+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=6, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.736 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-18T11:53:29.091408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=29, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.736 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-18T11:53:29.091408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=29, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.737 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-18T11:53:41.166841+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=41, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:12.737 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T12:09:32.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:33.037 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:33.039 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:33.049 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":540,"stamp":"2024-09-18T12:09:31.353758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3721,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178665,"num_write_kb":325391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21366,"num_bytes_recovered":7114771,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160707,"ondisk_log_size":160707,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1606440,"kb_used_data":66832,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281509080,"statfs":{"total":289910292480,"available":288265297920,"internally_reserved":0,"allocated":68435968,"data_stored":67411828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003322"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26849,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:24.537827+0000","last_change":"2024-09-18T12:09:24.537799+0000","last_active":"2024-09-18T12:09:24.537827+0000","last_peered":"2024-09-18T12:09:24.537827+0000","last_clean":"2024-09-18T12:09:24.537827+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:09:24.537827+0000","last_undegraded":"2024-09-18T12:09:24.537827+0000","last_fullsized":"2024-09-18T12:09:24.537827+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19536","last_scrub_stamp":"2024-09-18T12:09:24.537705+0000","last_deep_scrub":"43'19536","last_deep_scrub_stamp":"2024-09-18T12:09:24.537705+0000","last_clean_scrub_stamp":"2024-09-18T12:09:24.537705+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:39:37.019826+0000","scrub_duration":0.018119113999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19623,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:20.001673+0000","last_change":"2024-09-18T12:09:20.001638+0000","last_active":"2024-09-18T12:09:20.001673+0000","last_peered":"2024-09-18T12:09:20.001673+0000","last_clean":"2024-09-18T12:09:20.001673+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:09:20.001673+0000","last_undegraded":"2024-09-18T12:09:20.001673+0000","last_fullsized":"2024-09-18T12:09:20.001673+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19432","last_scrub_stamp":"2024-09-18T12:09:20.001512+0000","last_deep_scrub":"43'19432","last_deep_scrub_stamp":"2024-09-18T12:09:20.001512+0000","last_clean_scrub_stamp":"2024-09-18T12:09:20.001512+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:45:21.913079+0000","scrub_duration":0.021218435000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19772,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.978602+0000","last_change":"2024-09-18T12:06:26.993788+0000","last_active":"2024-09-18T12:09:03.978602+0000","last_peered":"2024-09-18T12:09:03.978602+0000","last_clean":"2024-09-18T12:09:03.978602+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:09:03.978602+0000","last_undegraded":"2024-09-18T12:09:03.978602+0000","last_fullsized":"2024-09-18T12:09:03.978602+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3208","last_scrub_stamp":"2024-09-18T11:54:55.665140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:54:55.665140+0000","objects_scrubbed":3138,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for deep scrub","scrub_duration":2.0122986329999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19322,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:19.024131+0000","last_change":"2024-09-18T12:09:19.024102+0000","last_active":"2024-09-18T12:09:19.024131+0000","last_peered":"2024-09-18T12:09:19.024131+0000","last_clean":"2024-09-18T12:09:19.024131+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:09:19.024131+0000","last_undegraded":"2024-09-18T12:09:19.024131+0000","last_fullsized":"2024-09-18T12:09:19.024131+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19260","last_scrub_stamp":"2024-09-18T12:09:19.023991+0000","last_deep_scrub":"43'19260","last_deep_scrub_stamp":"2024-09-18T12:09:19.023991+0000","last_clean_scrub_stamp":"2024-09-18T12:09:19.023991+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:21:16.952542+0000","scrub_duration":0.022296046,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19671,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:22.525305+0000","last_change":"2024-09-18T12:09:22.525276+0000","last_active":"2024-09-18T12:09:22.525305+0000","last_peered":"2024-09-18T12:09:22.525305+0000","last_clean":"2024-09-18T12:09:22.525305+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:09:22.525305+0000","last_undegraded":"2024-09-18T12:09:22.525305+0000","last_fullsized":"2024-09-18T12:09:22.525305+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19447","last_scrub_stamp":"2024-09-18T12:09:22.525180+0000","last_deep_scrub":"43'19447","last_deep_scrub_stamp":"2024-09-18T12:09:22.525180+0000","last_clean_scrub_stamp":"2024-09-18T12:09:22.525180+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:38:19.804402+0000","scrub_duration":0.022304794999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19524,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.978442+0000","last_change":"2024-09-18T12:08:34.977085+0000","last_active":"2024-09-18T12:09:03.978442+0000","last_peered":"2024-09-18T12:09:03.978442+0000","last_clean":"2024-09-18T12:09:03.978442+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:09:03.978442+0000","last_undegraded":"2024-09-18T12:09:03.978442+0000","last_fullsized":"2024-09-18T12:09:03.978442+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19421,"num_write_kb":38839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19482,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.978442+0000","last_change":"2024-09-18T12:07:25.661526+0000","last_active":"2024-09-18T12:09:03.978442+0000","last_peered":"2024-09-18T12:09:03.978442+0000","last_clean":"2024-09-18T12:09:03.978442+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:09:03.978442+0000","last_undegraded":"2024-09-18T12:09:03.978442+0000","last_fullsized":"2024-09-18T12:09:03.978442+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1234,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19370,"num_write_kb":38736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3564,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19860,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:16.610341+0000","last_change":"2024-09-18T12:09:16.610289+0000","last_active":"2024-09-18T12:09:16.610341+0000","last_peered":"2024-09-18T12:09:16.610341+0000","last_clean":"2024-09-18T12:09:16.610341+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:09:16.610341+0000","last_undegraded":"2024-09-18T12:09:16.610341+0000","last_fullsized":"2024-09-18T12:09:16.610341+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19602","last_scrub_stamp":"2024-09-18T12:09:16.610138+0000","last_deep_scrub":"43'19602","last_deep_scrub_stamp":"2024-09-18T12:09:16.610138+0000","last_clean_scrub_stamp":"2024-09-18T12:09:16.610138+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:40:44.523841+0000","scrub_duration":0.023653516999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:12.552818+0000","last_change":"2024-09-18T12:09:12.552784+0000","last_active":"2024-09-18T12:09:12.552818+0000","last_peered":"2024-09-18T12:09:12.552818+0000","last_clean":"2024-09-18T12:09:12.552818+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:09:12.552818+0000","last_undegraded":"2024-09-18T12:09:12.552818+0000","last_fullsized":"2024-09-18T12:09:12.552818+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:09:12.552670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:09:12.552670+0000","last_clean_scrub_stamp":"2024-09-18T12:09:12.552670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:31:23.043568+0000","scrub_duration":0.0021881169999999998,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26576,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:21.596607+0000","last_change":"2024-09-18T12:09:21.596579+0000","last_active":"2024-09-18T12:09:21.596607+0000","last_peered":"2024-09-18T12:09:21.596607+0000","last_clean":"2024-09-18T12:09:21.596607+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:09:21.596607+0000","last_undegraded":"2024-09-18T12:09:21.596607+0000","last_fullsized":"2024-09-18T12:09:21.596607+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19405","last_scrub_stamp":"2024-09-18T12:09:21.596480+0000","last_deep_scrub":"43'19405","last_deep_scrub_stamp":"2024-09-18T12:09:21.596480+0000","last_clean_scrub_stamp":"2024-09-18T12:09:21.596480+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:26:12.258093+0000","scrub_duration":0.020736787999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:22.900675+0000","last_change":"2024-09-18T12:09:22.900608+0000","last_active":"2024-09-18T12:09:22.900675+0000","last_peered":"2024-09-18T12:09:22.900675+0000","last_clean":"2024-09-18T12:09:22.900675+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:09:22.900675+0000","last_undegraded":"2024-09-18T12:09:22.900675+0000","last_fullsized":"2024-09-18T12:09:22.900675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:09:22.900409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:09:22.900409+0000","last_clean_scrub_stamp":"2024-09-18T12:09:22.900409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:58:58.351989+0000","scrub_duration":0.0030261580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26646,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:26.555640+0000","last_change":"2024-09-18T12:09:26.555612+0000","last_active":"2024-09-18T12:09:26.555640+0000","last_peered":"2024-09-18T12:09:26.555640+0000","last_clean":"2024-09-18T12:09:26.555640+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:09:26.555640+0000","last_undegraded":"2024-09-18T12:09:26.555640+0000","last_fullsized":"2024-09-18T12:09:26.555640+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19446","last_scrub_stamp":"2024-09-18T12:09:26.555517+0000","last_deep_scrub":"43'19446","last_deep_scrub_stamp":"2024-09-18T12:09:26.555517+0000","last_clean_scrub_stamp":"2024-09-18T12:09:26.555517+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:48:56.912168+0000","scrub_duration":0.019605145000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"61'3","reported_seq":125,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:11.663616+0000","last_change":"2024-09-18T12:09:11.663543+0000","last_active":"2024-09-18T12:09:11.663616+0000","last_peered":"2024-09-18T12:09:11.663616+0000","last_clean":"2024-09-18T12:09:11.663616+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:09:11.663616+0000","last_undegraded":"2024-09-18T12:09:11.663616+0000","last_fullsized":"2024-09-18T12:09:11.663616+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-18T12:09:11.663398+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-18T12:09:11.663398+0000","last_clean_scrub_stamp":"2024-09-18T12:09:11.663398+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:22:39.524143+0000","scrub_duration":0.0059491989999999996,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26808,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:03.976501+0000","last_change":"2024-09-18T12:09:03.055754+0000","last_active":"2024-09-18T12:09:03.976501+0000","last_peered":"2024-09-18T12:09:03.976501+0000","last_clean":"2024-09-18T12:09:03.976501+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:09:03.976501+0000","last_undegraded":"2024-09-18T12:09:03.976501+0000","last_fullsized":"2024-09-18T12:09:03.976501+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19463","last_scrub_stamp":"2024-09-18T12:09:03.055575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:03.055575+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.016902436999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"61'2","reported_seq":105,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:19.614032+0000","last_change":"2024-09-18T12:09:19.613965+0000","last_active":"2024-09-18T12:09:19.614032+0000","last_peered":"2024-09-18T12:09:19.614032+0000","last_clean":"2024-09-18T12:09:19.614032+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:09:19.614032+0000","last_undegraded":"2024-09-18T12:09:19.614032+0000","last_fullsized":"2024-09-18T12:09:19.614032+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T12:09:19.613650+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-18T12:09:19.613650+0000","last_clean_scrub_stamp":"2024-09-18T12:09:19.613650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:45:53.072929+0000","scrub_duration":0.0058578170000000004,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26944,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:17.986654+0000","last_change":"2024-09-18T12:09:17.986620+0000","last_active":"2024-09-18T12:09:17.986654+0000","last_peered":"2024-09-18T12:09:17.986654+0000","last_clean":"2024-09-18T12:09:17.986654+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:09:17.986654+0000","last_undegraded":"2024-09-18T12:09:17.986654+0000","last_fullsized":"2024-09-18T12:09:17.986654+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19498","last_scrub_stamp":"2024-09-18T12:09:17.986536+0000","last_deep_scrub":"43'19498","last_deep_scrub_stamp":"2024-09-18T12:09:17.986536+0000","last_clean_scrub_stamp":"2024-09-18T12:09:17.986536+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:33:19.500122+0000","scrub_duration":0.021132904000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19628,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:04.059328+0000","last_change":"2024-09-18T12:09:04.058553+0000","last_active":"2024-09-18T12:09:04.059328+0000","last_peered":"2024-09-18T12:09:04.059328+0000","last_clean":"2024-09-18T12:09:04.059328+0000","last_became_active":"2024-09-18T12:09:03.980052+0000","last_became_peered":"2024-09-18T12:09:03.980052+0000","last_unstale":"2024-09-18T12:09:04.059328+0000","last_undegraded":"2024-09-18T12:09:04.059328+0000","last_fullsized":"2024-09-18T12:09:04.059328+0000","mapping_epoch":60,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T11:53:41.166841+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19527,"num_write_kb":39053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3538,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":88,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:14.631777+0000","last_change":"2024-09-18T12:09:14.631711+0000","last_active":"2024-09-18T12:09:14.631777+0000","last_peered":"2024-09-18T12:09:14.631777+0000","last_clean":"2024-09-18T12:09:14.631777+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:09:14.631777+0000","last_undegraded":"2024-09-18T12:09:14.631777+0000","last_fullsized":"2024-09-18T12:09:14.631777+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:09:14.631509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:09:14.631509+0000","last_clean_scrub_stamp":"2024-09-18T12:09:14.631509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:45:14.198290+0000","scrub_duration":0.0033924979999999999,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26711,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:22.566276+0000","last_change":"2024-09-18T12:09:22.566249+0000","last_active":"2024-09-18T12:09:22.566276+0000","last_peered":"2024-09-18T12:09:22.566276+0000","last_clean":"2024-09-18T12:09:22.566276+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:09:22.566276+0000","last_undegraded":"2024-09-18T12:09:22.566276+0000","last_fullsized":"2024-09-18T12:09:22.566276+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19522","last_scrub_stamp":"2024-09-18T12:09:22.566151+0000","last_deep_scrub":"43'19522","last_deep_scrub_stamp":"2024-09-18T12:09:22.566151+0000","last_clean_scrub_stamp":"2024-09-18T12:09:22.566151+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:13:54.377029+0000","scrub_duration":0.024776633999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":72,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:26.896127+0000","last_change":"2024-09-18T12:09:26.896000+0000","last_active":"2024-09-18T12:09:26.896127+0000","last_peered":"2024-09-18T12:09:26.896127+0000","last_clean":"2024-09-18T12:09:26.896127+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:09:26.896127+0000","last_undegraded":"2024-09-18T12:09:26.896127+0000","last_fullsized":"2024-09-18T12:09:26.896127+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:09:26.895197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:09:26.895197+0000","last_clean_scrub_stamp":"2024-09-18T12:09:26.895197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:59:48.186936+0000","scrub_duration":0.0030450210000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19319,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:06.520644+0000","last_change":"2024-09-18T12:09:06.520583+0000","last_active":"2024-09-18T12:09:06.520644+0000","last_peered":"2024-09-18T12:09:06.520644+0000","last_clean":"2024-09-18T12:09:06.520644+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:09:06.520644+0000","last_undegraded":"2024-09-18T12:09:06.520644+0000","last_fullsized":"2024-09-18T12:09:06.520644+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19198","last_scrub_stamp":"2024-09-18T12:09:06.520451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:06.520451+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.020014351999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":88,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:17.672684+0000","last_change":"2024-09-18T12:09:17.672626+0000","last_active":"2024-09-18T12:09:17.672684+0000","last_peered":"2024-09-18T12:09:17.672684+0000","last_clean":"2024-09-18T12:09:17.672684+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:09:17.672684+0000","last_undegraded":"2024-09-18T12:09:17.672684+0000","last_fullsized":"2024-09-18T12:09:17.672684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:09:17.672428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:09:17.672428+0000","last_clean_scrub_stamp":"2024-09-18T12:09:17.672428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:18:36.115701+0000","scrub_duration":0.0028719599999999998,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":99,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:13.601507+0000","last_change":"2024-09-18T12:09:13.601448+0000","last_active":"2024-09-18T12:09:13.601507+0000","last_peered":"2024-09-18T12:09:13.601507+0000","last_clean":"2024-09-18T12:09:13.601507+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:09:13.601507+0000","last_undegraded":"2024-09-18T12:09:13.601507+0000","last_fullsized":"2024-09-18T12:09:13.601507+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:09:13.601250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:09:13.601250+0000","last_clean_scrub_stamp":"2024-09-18T12:09:13.601250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:15:18.446589+0000","scrub_duration":0.0036649009999999999,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"43'19512","reported_seq":19631,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:07.536257+0000","last_change":"2024-09-18T12:09:07.536196+0000","last_active":"2024-09-18T12:09:07.536257+0000","last_peered":"2024-09-18T12:09:07.536257+0000","last_clean":"2024-09-18T12:09:07.536257+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:09:07.536257+0000","last_undegraded":"2024-09-18T12:09:07.536257+0000","last_fullsized":"2024-09-18T12:09:07.536257+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19512","last_scrub_stamp":"2024-09-18T12:09:07.536101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T11:53:41.166841+0000","last_clean_scrub_stamp":"2024-09-18T12:09:07.536101+0000","objects_scrubbed":1,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.020700907000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3702,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178663,"num_write_kb":325389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21365,"num_bytes_recovered":7114752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":66293760,"data_stored":66286802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051771,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":332972,"kb_used_data":22628,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94038868,"statfs":{"total":96636764160,"available":96295800832,"internally_reserved":0,"allocated":23171072,"data_stored":22837144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.58599999999999997,"15min":0.71199999999999997},"min":{"1min":0.26100000000000001,"5min":0.19700000000000001,"15min":0.153},"max":{"1min":0.58899999999999997,"5min":5.6840000000000002,"15min":19.776},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.63300000000000001,"15min":1.169},"min":{"1min":0.30399999999999999,"5min":0.187,"15min":0.17499999999999999},"max":{"1min":0.496,"5min":5.8029999999999999,"15min":26.702000000000002},"last":0.53400000000000003}]},{"osd":1,"last update":"Wed Sep 18 12:09:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52100000000000002,"15min":1.345},"min":{"1min":0.32700000000000001,"5min":0.28899999999999998,"15min":0.20999999999999999},"max":{"1min":0.753,"5min":5.1820000000000004,"15min":16.847999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.505,"15min":1.0089999999999999},"min":{"1min":0.32800000000000001,"5min":0.23799999999999999,"15min":0.20699999999999999},"max":{"1min":0.60099999999999998,"5min":5.415,"15min":12.712},"last":0.59899999999999998}]}]},{"osd":1,"up_from":35,"seq":150323855533,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919488,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452352,"statfs":{"total":96636764160,"available":95695208448,"internally_reserved":0,"allocated":516096,"data_stored":170990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.442},"min":{"1min":0.26100000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.58899999999999997,"5min":3.0859999999999999,"15min":8.6210000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55700000000000005,"15min":1.131},"min":{"1min":0.29499999999999998,"5min":0.193,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":8.3200000000000003,"15min":12.244999999999999},"last":0.46500000000000002}]},{"osd":2,"last update":"Wed Sep 18 12:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.72699999999999998,"15min":1.256},"min":{"1min":0.30599999999999999,"5min":0.20899999999999999,"15min":0.19600000000000001},"max":{"1min":0.54300000000000004,"5min":11.41,"15min":13.737},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48799999999999999,"15min":0.73799999999999999},"min":{"1min":0.32100000000000001,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":0.66500000000000004,"5min":4.3479999999999999,"15min":11.996},"last":0.44600000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247999,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":353980,"kb_used_data":43700,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94017860,"statfs":{"total":96636764160,"available":96274288640,"internally_reserved":0,"allocated":44748800,"data_stored":44403694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.503,"15min":0.90300000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.161},"max":{"1min":0.54400000000000004,"5min":5.1749999999999998,"15min":11.521000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.435,"15min":0.80400000000000005},"min":{"1min":0.312,"5min":0.189,"15min":0.189},"max":{"1min":0.55800000000000005,"5min":3.2440000000000002,"15min":8.3460000000000001},"last":0.46700000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.51700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43745280,"data_stored":43741801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22544384,"data_stored":22544384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:33.051 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-09-18T11:54:55.665140+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=54, tm_sec=55, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:33.052 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-09-18T11:53:41.166841+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=41, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:33.052 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-09-18T11:53:41.166841+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=41, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:33.052 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-18T12:09:03.055575+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:33.052 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-18T11:53:41.166841+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=11, tm_min=53, tm_sec=41, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:33.052 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-18T12:09:06.520451+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=6, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:33.053 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-18T12:09:07.536101+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=7, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=12, tm_min=9, tm_sec=10, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T12:09:33.053 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T12:09:53.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T12:09:53.352 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-18T12:09:53.353 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-18T12:09:53.365 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":550,"stamp":"2024-09-18T12:09:51.359175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":636,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120347,"num_write_kb":208763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10733,"num_bytes_recovered":7106579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160708,"ondisk_log_size":160708,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1589640,"kb_used_data":50032,"kb_used_omap":13,"kb_used_meta":1539570,"kb_avail":281525880,"statfs":{"total":289910292480,"available":288282501120,"internally_reserved":0,"allocated":51232768,"data_stored":50208628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13806,"internal_metadata":1576520210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3085,"num_objects":-5,"num_object_clones":0,"num_object_copies":-10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-58318,"num_write_kb":-116628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-10633,"num_bytes_recovered":-8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003236"},"pg_stats":[{"pgid":"2.f","version":"43'19536","reported_seq":26849,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:24.537827+0000","last_change":"2024-09-18T12:09:24.537799+0000","last_active":"2024-09-18T12:09:24.537827+0000","last_peered":"2024-09-18T12:09:24.537827+0000","last_clean":"2024-09-18T12:09:24.537827+0000","last_became_active":"2024-09-18T12:07:56.664150+0000","last_became_peered":"2024-09-18T12:07:56.664150+0000","last_unstale":"2024-09-18T12:09:24.537827+0000","last_undegraded":"2024-09-18T12:09:24.537827+0000","last_fullsized":"2024-09-18T12:09:24.537827+0000","mapping_epoch":56,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19536","last_scrub_stamp":"2024-09-18T12:09:24.537705+0000","last_deep_scrub":"43'19536","last_deep_scrub_stamp":"2024-09-18T12:09:24.537705+0000","last_clean_scrub_stamp":"2024-09-18T12:09:24.537705+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:39:37.019826+0000","scrub_duration":0.018119113999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19536,"num_write_kb":39072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"43'19432","reported_seq":19623,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:20.001673+0000","last_change":"2024-09-18T12:09:20.001638+0000","last_active":"2024-09-18T12:09:20.001673+0000","last_peered":"2024-09-18T12:09:20.001673+0000","last_clean":"2024-09-18T12:09:20.001673+0000","last_became_active":"2024-09-18T11:54:18.439848+0000","last_became_peered":"2024-09-18T11:54:18.439848+0000","last_unstale":"2024-09-18T12:09:20.001673+0000","last_undegraded":"2024-09-18T12:09:20.001673+0000","last_fullsized":"2024-09-18T12:09:20.001673+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19432","last_scrub_stamp":"2024-09-18T12:09:20.001512+0000","last_deep_scrub":"43'19432","last_deep_scrub_stamp":"2024-09-18T12:09:20.001512+0000","last_clean_scrub_stamp":"2024-09-18T12:09:20.001512+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:45:21.913079+0000","scrub_duration":0.021218435000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16624,"num_write_kb":27644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"43'19558","reported_seq":19782,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:28.533810+0000","last_change":"2024-09-18T12:09:28.533769+0000","last_active":"2024-09-18T12:09:28.533810+0000","last_peered":"2024-09-18T12:09:28.533810+0000","last_clean":"2024-09-18T12:09:28.533810+0000","last_became_active":"2024-09-18T12:06:26.871862+0000","last_became_peered":"2024-09-18T12:06:26.871862+0000","last_unstale":"2024-09-18T12:09:28.533810+0000","last_undegraded":"2024-09-18T12:09:28.533810+0000","last_fullsized":"2024-09-18T12:09:28.533810+0000","mapping_epoch":50,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19558","last_scrub_stamp":"2024-09-18T12:09:28.533634+0000","last_deep_scrub":"43'19558","last_deep_scrub_stamp":"2024-09-18T12:09:28.533634+0000","last_clean_scrub_stamp":"2024-09-18T12:09:28.533634+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:53:40.701423+0000","scrub_duration":0.019491929000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16416,"num_write_kb":26553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3537,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"43'19260","reported_seq":19322,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:19.024131+0000","last_change":"2024-09-18T12:09:19.024102+0000","last_active":"2024-09-18T12:09:19.024131+0000","last_peered":"2024-09-18T12:09:19.024131+0000","last_clean":"2024-09-18T12:09:19.024131+0000","last_became_active":"2024-09-18T11:54:18.438767+0000","last_became_peered":"2024-09-18T11:54:18.438767+0000","last_unstale":"2024-09-18T12:09:19.024131+0000","last_undegraded":"2024-09-18T12:09:19.024131+0000","last_fullsized":"2024-09-18T12:09:19.024131+0000","mapping_epoch":29,"log_start":"39'9200","ondisk_log_start":"39'9200","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19260","last_scrub_stamp":"2024-09-18T12:09:19.023991+0000","last_deep_scrub":"43'19260","last_deep_scrub_stamp":"2024-09-18T12:09:19.023991+0000","last_clean_scrub_stamp":"2024-09-18T12:09:19.023991+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:21:16.952542+0000","scrub_duration":0.022296046,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19260,"num_write_kb":38520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"43'19447","reported_seq":19671,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:22.525305+0000","last_change":"2024-09-18T12:09:22.525276+0000","last_active":"2024-09-18T12:09:22.525305+0000","last_peered":"2024-09-18T12:09:22.525305+0000","last_clean":"2024-09-18T12:09:22.525305+0000","last_became_active":"2024-09-18T11:54:18.440982+0000","last_became_peered":"2024-09-18T11:54:18.440982+0000","last_unstale":"2024-09-18T12:09:22.525305+0000","last_undegraded":"2024-09-18T12:09:22.525305+0000","last_fullsized":"2024-09-18T12:09:22.525305+0000","mapping_epoch":29,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19447","last_scrub_stamp":"2024-09-18T12:09:22.525180+0000","last_deep_scrub":"43'19447","last_deep_scrub_stamp":"2024-09-18T12:09:22.525180+0000","last_clean_scrub_stamp":"2024-09-18T12:09:22.525180+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:38:19.804402+0000","scrub_duration":0.022304794999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16612,"num_write_kb":27568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"43'19423","reported_seq":19534,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:41.587138+0000","last_change":"2024-09-18T12:09:41.587110+0000","last_active":"2024-09-18T12:09:41.587138+0000","last_peered":"2024-09-18T12:09:41.587138+0000","last_clean":"2024-09-18T12:09:41.587138+0000","last_became_active":"2024-09-18T12:08:34.840361+0000","last_became_peered":"2024-09-18T12:08:34.840361+0000","last_unstale":"2024-09-18T12:09:41.587138+0000","last_undegraded":"2024-09-18T12:09:41.587138+0000","last_fullsized":"2024-09-18T12:09:41.587138+0000","mapping_epoch":58,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19423","last_scrub_stamp":"2024-09-18T12:09:41.587016+0000","last_deep_scrub":"43'19423","last_deep_scrub_stamp":"2024-09-18T12:09:41.587016+0000","last_clean_scrub_stamp":"2024-09-18T12:09:41.587016+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:07:17.613087+0000","scrub_duration":0.017856552000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"43'19372","reported_seq":19492,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:40.538763+0000","last_change":"2024-09-18T12:09:40.538736+0000","last_active":"2024-09-18T12:09:40.538763+0000","last_peered":"2024-09-18T12:09:40.538763+0000","last_clean":"2024-09-18T12:09:40.538763+0000","last_became_active":"2024-09-18T12:07:25.550527+0000","last_became_peered":"2024-09-18T12:07:25.550527+0000","last_unstale":"2024-09-18T12:09:40.538763+0000","last_undegraded":"2024-09-18T12:09:40.538763+0000","last_fullsized":"2024-09-18T12:09:40.538763+0000","mapping_epoch":54,"log_start":"39'9300","ondisk_log_start":"39'9300","created":14,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19372","last_scrub_stamp":"2024-09-18T12:09:40.538630+0000","last_deep_scrub":"43'19372","last_deep_scrub_stamp":"2024-09-18T12:09:40.538630+0000","last_clean_scrub_stamp":"2024-09-18T12:09:40.538630+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:09:00.888817+0000","scrub_duration":0.016696111999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"43'19602","reported_seq":19860,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:16.610341+0000","last_change":"2024-09-18T12:09:16.610289+0000","last_active":"2024-09-18T12:09:16.610341+0000","last_peered":"2024-09-18T12:09:16.610341+0000","last_clean":"2024-09-18T12:09:16.610341+0000","last_became_active":"2024-09-18T11:54:18.445785+0000","last_became_peered":"2024-09-18T11:54:18.445785+0000","last_unstale":"2024-09-18T12:09:16.610341+0000","last_undegraded":"2024-09-18T12:09:16.610341+0000","last_fullsized":"2024-09-18T12:09:16.610341+0000","mapping_epoch":29,"log_start":"39'9600","ondisk_log_start":"39'9600","created":14,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19602","last_scrub_stamp":"2024-09-18T12:09:16.610138+0000","last_deep_scrub":"43'19602","last_deep_scrub_stamp":"2024-09-18T12:09:16.610138+0000","last_clean_scrub_stamp":"2024-09-18T12:09:16.610138+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:40:44.523841+0000","scrub_duration":0.023653516999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15919,"num_write_kb":24476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:12.552818+0000","last_change":"2024-09-18T12:09:12.552784+0000","last_active":"2024-09-18T12:09:12.552818+0000","last_peered":"2024-09-18T12:09:12.552818+0000","last_clean":"2024-09-18T12:09:12.552818+0000","last_became_active":"2024-09-18T11:55:04.263987+0000","last_became_peered":"2024-09-18T11:55:04.263987+0000","last_unstale":"2024-09-18T12:09:12.552818+0000","last_undegraded":"2024-09-18T12:09:12.552818+0000","last_fullsized":"2024-09-18T12:09:12.552818+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:09:12.552670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:09:12.552670+0000","last_clean_scrub_stamp":"2024-09-18T12:09:12.552670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:31:23.043568+0000","scrub_duration":0.0021881169999999998,"objects_trimmed":0,"snaptrim_duration":0.001116389,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"43'19405","reported_seq":26576,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:21.596607+0000","last_change":"2024-09-18T12:09:21.596579+0000","last_active":"2024-09-18T12:09:21.596607+0000","last_peered":"2024-09-18T12:09:21.596607+0000","last_clean":"2024-09-18T12:09:21.596607+0000","last_became_active":"2024-09-18T11:56:57.499312+0000","last_became_peered":"2024-09-18T11:56:57.499312+0000","last_unstale":"2024-09-18T12:09:21.596607+0000","last_undegraded":"2024-09-18T12:09:21.596607+0000","last_fullsized":"2024-09-18T12:09:21.596607+0000","mapping_epoch":38,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19405","last_scrub_stamp":"2024-09-18T12:09:21.596480+0000","last_deep_scrub":"43'19405","last_deep_scrub_stamp":"2024-09-18T12:09:21.596480+0000","last_clean_scrub_stamp":"2024-09-18T12:09:21.596480+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:26:12.258093+0000","scrub_duration":0.020736787999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:22.900675+0000","last_change":"2024-09-18T12:09:22.900608+0000","last_active":"2024-09-18T12:09:22.900675+0000","last_peered":"2024-09-18T12:09:22.900675+0000","last_clean":"2024-09-18T12:09:22.900675+0000","last_became_active":"2024-09-18T11:54:18.438817+0000","last_became_peered":"2024-09-18T11:54:18.438817+0000","last_unstale":"2024-09-18T12:09:22.900675+0000","last_undegraded":"2024-09-18T12:09:22.900675+0000","last_fullsized":"2024-09-18T12:09:22.900675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:09:22.900409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:09:22.900409+0000","last_clean_scrub_stamp":"2024-09-18T12:09:22.900409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:58:58.351989+0000","scrub_duration":0.0030261580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"43'19446","reported_seq":26646,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:26.555640+0000","last_change":"2024-09-18T12:09:26.555612+0000","last_active":"2024-09-18T12:09:26.555640+0000","last_peered":"2024-09-18T12:09:26.555640+0000","last_clean":"2024-09-18T12:09:26.555640+0000","last_became_active":"2024-09-18T12:04:39.157085+0000","last_became_peered":"2024-09-18T12:04:39.157085+0000","last_unstale":"2024-09-18T12:09:26.555640+0000","last_undegraded":"2024-09-18T12:09:26.555640+0000","last_fullsized":"2024-09-18T12:09:26.555640+0000","mapping_epoch":44,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19446","last_scrub_stamp":"2024-09-18T12:09:26.555517+0000","last_deep_scrub":"43'19446","last_deep_scrub_stamp":"2024-09-18T12:09:26.555517+0000","last_clean_scrub_stamp":"2024-09-18T12:09:26.555517+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:48:56.912168+0000","scrub_duration":0.019605145000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"61'3","reported_seq":125,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:11.663616+0000","last_change":"2024-09-18T12:09:11.663543+0000","last_active":"2024-09-18T12:09:11.663616+0000","last_peered":"2024-09-18T12:09:11.663616+0000","last_clean":"2024-09-18T12:09:11.663616+0000","last_became_active":"2024-09-18T11:55:04.714970+0000","last_became_peered":"2024-09-18T11:55:04.714970+0000","last_unstale":"2024-09-18T12:09:11.663616+0000","last_undegraded":"2024-09-18T12:09:11.663616+0000","last_fullsized":"2024-09-18T12:09:11.663616+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'3","last_scrub_stamp":"2024-09-18T12:09:11.663398+0000","last_deep_scrub":"61'3","last_deep_scrub_stamp":"2024-09-18T12:09:11.663398+0000","last_clean_scrub_stamp":"2024-09-18T12:09:11.663398+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:22:39.524143+0000","scrub_duration":0.0059491989999999996,"objects_trimmed":0,"snaptrim_duration":0.00073140300000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"43'19463","reported_seq":26818,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:38.968472+0000","last_change":"2024-09-18T12:09:38.968436+0000","last_active":"2024-09-18T12:09:38.968472+0000","last_peered":"2024-09-18T12:09:38.968472+0000","last_clean":"2024-09-18T12:09:38.968472+0000","last_became_active":"2024-09-18T12:05:18.501510+0000","last_became_peered":"2024-09-18T12:05:18.501510+0000","last_unstale":"2024-09-18T12:09:38.968472+0000","last_undegraded":"2024-09-18T12:09:38.968472+0000","last_fullsized":"2024-09-18T12:09:38.968472+0000","mapping_epoch":46,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19463","last_scrub_stamp":"2024-09-18T12:09:38.968321+0000","last_deep_scrub":"43'19463","last_deep_scrub_stamp":"2024-09-18T12:09:38.968321+0000","last_clean_scrub_stamp":"2024-09-18T12:09:38.968321+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:48:45.619917+0000","scrub_duration":0.018322801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"61'2","reported_seq":105,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:19.614032+0000","last_change":"2024-09-18T12:09:19.613965+0000","last_active":"2024-09-18T12:09:19.614032+0000","last_peered":"2024-09-18T12:09:19.614032+0000","last_clean":"2024-09-18T12:09:19.614032+0000","last_became_active":"2024-09-18T11:55:04.343459+0000","last_became_peered":"2024-09-18T11:55:04.343459+0000","last_unstale":"2024-09-18T12:09:19.614032+0000","last_undegraded":"2024-09-18T12:09:19.614032+0000","last_fullsized":"2024-09-18T12:09:19.614032+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T12:09:19.613650+0000","last_deep_scrub":"61'2","last_deep_scrub_stamp":"2024-09-18T12:09:19.613650+0000","last_clean_scrub_stamp":"2024-09-18T12:09:19.613650+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:45:53.072929+0000","scrub_duration":0.0058578170000000004,"objects_trimmed":0,"snaptrim_duration":0.00092017399999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"43'19498","reported_seq":26944,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:17.986654+0000","last_change":"2024-09-18T12:09:17.986620+0000","last_active":"2024-09-18T12:09:17.986654+0000","last_peered":"2024-09-18T12:09:17.986654+0000","last_clean":"2024-09-18T12:09:17.986654+0000","last_became_active":"2024-09-18T12:01:44.514394+0000","last_became_peered":"2024-09-18T12:01:44.514394+0000","last_unstale":"2024-09-18T12:09:17.986654+0000","last_undegraded":"2024-09-18T12:09:17.986654+0000","last_fullsized":"2024-09-18T12:09:17.986654+0000","mapping_epoch":42,"log_start":"39'9400","ondisk_log_start":"39'9400","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19498","last_scrub_stamp":"2024-09-18T12:09:17.986536+0000","last_deep_scrub":"43'19498","last_deep_scrub_stamp":"2024-09-18T12:09:17.986536+0000","last_clean_scrub_stamp":"2024-09-18T12:09:17.986536+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:33:19.500122+0000","scrub_duration":0.021132904000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15978,"num_write_kb":24928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"43'19528","reported_seq":19638,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:44.569242+0000","last_change":"2024-09-18T12:09:44.569199+0000","last_active":"2024-09-18T12:09:44.569242+0000","last_peered":"2024-09-18T12:09:44.569242+0000","last_clean":"2024-09-18T12:09:44.569242+0000","last_became_active":"2024-09-18T12:09:03.980052+0000","last_became_peered":"2024-09-18T12:09:03.980052+0000","last_unstale":"2024-09-18T12:09:44.569242+0000","last_undegraded":"2024-09-18T12:09:44.569242+0000","last_fullsized":"2024-09-18T12:09:44.569242+0000","mapping_epoch":60,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19528","last_scrub_stamp":"2024-09-18T12:09:44.569108+0000","last_deep_scrub":"43'19528","last_deep_scrub_stamp":"2024-09-18T12:09:44.569108+0000","last_clean_scrub_stamp":"2024-09-18T12:09:44.569108+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:08:48.209030+0000","scrub_duration":0.018787844000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":88,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:14.631777+0000","last_change":"2024-09-18T12:09:14.631711+0000","last_active":"2024-09-18T12:09:14.631777+0000","last_peered":"2024-09-18T12:09:14.631777+0000","last_clean":"2024-09-18T12:09:14.631777+0000","last_became_active":"2024-09-18T11:55:04.451384+0000","last_became_peered":"2024-09-18T11:55:04.451384+0000","last_unstale":"2024-09-18T12:09:14.631777+0000","last_undegraded":"2024-09-18T12:09:14.631777+0000","last_fullsized":"2024-09-18T12:09:14.631777+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:09:14.631509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:09:14.631509+0000","last_clean_scrub_stamp":"2024-09-18T12:09:14.631509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:45:14.198290+0000","scrub_duration":0.0033924979999999999,"objects_trimmed":0,"snaptrim_duration":0.0014231249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"43'19522","reported_seq":26711,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:22.566276+0000","last_change":"2024-09-18T12:09:22.566249+0000","last_active":"2024-09-18T12:09:22.566276+0000","last_peered":"2024-09-18T12:09:22.566276+0000","last_clean":"2024-09-18T12:09:22.566276+0000","last_became_active":"2024-09-18T11:59:16.725155+0000","last_became_peered":"2024-09-18T11:59:16.725155+0000","last_unstale":"2024-09-18T12:09:22.566276+0000","last_undegraded":"2024-09-18T12:09:22.566276+0000","last_fullsized":"2024-09-18T12:09:22.566276+0000","mapping_epoch":40,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19522","last_scrub_stamp":"2024-09-18T12:09:22.566151+0000","last_deep_scrub":"43'19522","last_deep_scrub_stamp":"2024-09-18T12:09:22.566151+0000","last_clean_scrub_stamp":"2024-09-18T12:09:22.566151+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:13:54.377029+0000","scrub_duration":0.024776633999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":72,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:26.896127+0000","last_change":"2024-09-18T12:09:26.896000+0000","last_active":"2024-09-18T12:09:26.896127+0000","last_peered":"2024-09-18T12:09:26.896127+0000","last_clean":"2024-09-18T12:09:26.896127+0000","last_became_active":"2024-09-18T11:54:18.440016+0000","last_became_peered":"2024-09-18T11:54:18.440016+0000","last_unstale":"2024-09-18T12:09:26.896127+0000","last_undegraded":"2024-09-18T12:09:26.896127+0000","last_fullsized":"2024-09-18T12:09:26.896127+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:09:26.895197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:09:26.895197+0000","last_clean_scrub_stamp":"2024-09-18T12:09:26.895197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:59:48.186936+0000","scrub_duration":0.0030450210000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"43'19198","reported_seq":19329,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:27.560940+0000","last_change":"2024-09-18T12:09:27.560909+0000","last_active":"2024-09-18T12:09:27.560940+0000","last_peered":"2024-09-18T12:09:27.560940+0000","last_clean":"2024-09-18T12:09:27.560940+0000","last_became_active":"2024-09-18T12:05:48.672224+0000","last_became_peered":"2024-09-18T12:05:48.672224+0000","last_unstale":"2024-09-18T12:09:27.560940+0000","last_undegraded":"2024-09-18T12:09:27.560940+0000","last_fullsized":"2024-09-18T12:09:27.560940+0000","mapping_epoch":48,"log_start":"39'9100","ondisk_log_start":"39'9100","created":14,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"43'19198","last_scrub_stamp":"2024-09-18T12:09:27.560815+0000","last_deep_scrub":"43'19198","last_deep_scrub_stamp":"2024-09-18T12:09:27.560815+0000","last_clean_scrub_stamp":"2024-09-18T12:09:27.560815+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:31:29.237726+0000","scrub_duration":0.019736897999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":88,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:17.672684+0000","last_change":"2024-09-18T12:09:17.672626+0000","last_active":"2024-09-18T12:09:17.672684+0000","last_peered":"2024-09-18T12:09:17.672684+0000","last_clean":"2024-09-18T12:09:17.672684+0000","last_became_active":"2024-09-18T11:55:04.427439+0000","last_became_peered":"2024-09-18T11:55:04.427439+0000","last_unstale":"2024-09-18T12:09:17.672684+0000","last_undegraded":"2024-09-18T12:09:17.672684+0000","last_fullsized":"2024-09-18T12:09:17.672684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:09:17.672428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:09:17.672428+0000","last_clean_scrub_stamp":"2024-09-18T12:09:17.672428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:18:36.115701+0000","scrub_duration":0.0028719599999999998,"objects_trimmed":0,"snaptrim_duration":0.001470521,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":99,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:13.601507+0000","last_change":"2024-09-18T12:09:13.601448+0000","last_active":"2024-09-18T12:09:13.601507+0000","last_peered":"2024-09-18T12:09:13.601507+0000","last_clean":"2024-09-18T12:09:13.601507+0000","last_became_active":"2024-09-18T11:55:04.255734+0000","last_became_peered":"2024-09-18T11:55:04.255734+0000","last_unstale":"2024-09-18T12:09:13.601507+0000","last_undegraded":"2024-09-18T12:09:13.601507+0000","last_fullsized":"2024-09-18T12:09:13.601507+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T12:09:13.601250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T12:09:13.601250+0000","last_clean_scrub_stamp":"2024-09-18T12:09:13.601250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:15:18.446589+0000","scrub_duration":0.0036649009999999999,"objects_trimmed":0,"snaptrim_duration":0.001513545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"61'19513","reported_seq":19642,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-18T12:09:38.607229+0000","last_change":"2024-09-18T12:09:38.607173+0000","last_active":"2024-09-18T12:09:38.607229+0000","last_peered":"2024-09-18T12:09:38.607229+0000","last_clean":"2024-09-18T12:09:38.607229+0000","last_became_active":"2024-09-18T12:06:56.105073+0000","last_became_peered":"2024-09-18T12:06:56.105073+0000","last_unstale":"2024-09-18T12:09:38.607229+0000","last_undegraded":"2024-09-18T12:09:38.607229+0000","last_fullsized":"2024-09-18T12:09:38.607229+0000","mapping_epoch":52,"log_start":"39'9500","ondisk_log_start":"39'9500","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'19513","last_scrub_stamp":"2024-09-18T12:09:38.606981+0000","last_deep_scrub":"61'19513","last_deep_scrub_stamp":"2024-09-18T12:09:38.606981+0000","last_clean_scrub_stamp":"2024-09-18T12:09:38.606981+0000","objects_scrubbed":1,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:59:59.110753+0000","scrub_duration":0.031141353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120345,"num_write_kb":208761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10732,"num_bytes_recovered":7106560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49213440,"data_stored":49206482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160703,"ondisk_log_size":160703,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":29,"seq":124554051775,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325772,"kb_used_data":15428,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94046068,"statfs":{"total":96636764160,"available":96303173632,"internally_reserved":0,"allocated":15798272,"data_stored":15464344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317779462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.40500000000000003,"15min":0.71899999999999997},"min":{"1min":0.33100000000000002,"5min":0.23100000000000001,"15min":0.153},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":19.776},"last":0.443},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.441,"15min":1.1839999999999999},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.17499999999999999},"max":{"1min":0.67200000000000004,"5min":0.68300000000000005,"15min":26.702000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Wed Sep 18 12:09:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52100000000000002,"15min":1.345},"min":{"1min":0.32700000000000001,"5min":0.28899999999999998,"15min":0.20999999999999999},"max":{"1min":0.753,"5min":5.1820000000000004,"15min":16.847999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.505,"15min":1.0089999999999999},"min":{"1min":0.32800000000000001,"5min":0.23799999999999999,"15min":0.20699999999999999},"max":{"1min":0.60099999999999998,"5min":5.415,"15min":12.712},"last":0.56200000000000006}]}]},{"osd":1,"up_from":35,"seq":150323855537,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919488,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":918971,"kb_avail":93452352,"statfs":{"total":96636764160,"available":95695208448,"internally_reserved":0,"allocated":516096,"data_stored":170990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":941026822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 12:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39900000000000002,"15min":0.44},"min":{"1min":0.23200000000000001,"5min":0.183,"15min":0.17399999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":8.6210000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.41599999999999998,"15min":1.0960000000000001},"min":{"1min":0.249,"5min":0.23100000000000001,"15min":0.185},"max":{"1min":0.59599999999999997,"5min":0.63500000000000001,"15min":12.244999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Wed Sep 18 12:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.42799999999999999,"15min":1.228},"min":{"1min":0.28799999999999998,"5min":0.24299999999999999,"15min":0.19600000000000001},"max":{"1min":0.63700000000000001,"5min":0.65700000000000003,"15min":13.737},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.47099999999999997,"15min":0.73299999999999998},"min":{"1min":0.33900000000000002,"5min":0.28499999999999998,"15min":0.20999999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":11.996},"last":0.61199999999999999}]}]},{"osd":0,"up_from":23,"seq":98784248003,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344380,"kb_used_data":34100,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":94027460,"statfs":{"total":96636764160,"available":96284119040,"internally_reserved":0,"allocated":34918400,"data_stored":34573294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":317713926},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 12:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.40500000000000003,"15min":0.86499999999999999},"min":{"1min":0.30399999999999999,"5min":0.215,"15min":0.161},"max":{"1min":0.60399999999999998,"5min":0.71399999999999997,"15min":11.521000000000001},"last":0.441},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.42399999999999999,"15min":0.75900000000000001},"min":{"1min":0.30499999999999999,"5min":0.249,"15min":0.189},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":8.3460000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Wed Sep 18 12:08:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.55300000000000005,"15min":1.127},"min":{"1min":0.34999999999999998,"5min":0.21299999999999999,"15min":0.16500000000000001},"max":{"1min":0.58799999999999997,"5min":8.8650000000000002,"15min":13.773},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.68400000000000005,"15min":0.98999999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.17499999999999999},"max":{"1min":0.63600000000000001,"5min":8.6430000000000007,"15min":11.957000000000001},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33914880,"data_stored":33911401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15294464,"data_stored":15294464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T12:09:53.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-18T12:09:53.682 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-18T12:09:53.682 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-18T12:09:53.682 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-18T12:09:53.683 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:09:58.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-18T12:09:59.784 INFO:tasks.ceph.osd.0:Stopped 2024-09-18T12:09:59.785 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-18T12:09:59.785 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:09:59.876 INFO:tasks.ceph.osd.1:Stopped 2024-09-18T12:09:59.877 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-18T12:09:59.877 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:10:03.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-18T12:10:03.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-18T12:10:05.978 INFO:tasks.ceph.osd.2:Stopped 2024-09-18T12:10:05.978 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-18T12:10:05.979 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-18T12:10:05.979 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:10:06.006 INFO:tasks.ceph.mgr.x:Stopped 2024-09-18T12:10:06.006 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-18T12:10:06.006 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T12:10:06.006 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:10:06.023 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T12:10:06.023 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T12:10:06.023 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:10:06.041 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T12:10:06.041 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T12:10:06.042 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T12:10:06.103 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T12:10:06.103 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-18T12:10:06.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-09-18T12:10:06.159 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-18T12:10:06.159 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-18T12:10:06.972 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-18T12:10:06.972 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-18T12:10:07.250 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-18T12:10:07.250 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-18T12:10:07.470 INFO:tasks.ceph:Archiving mon data... 2024-09-18T12:10:07.472 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910993/data/mon.a.tgz 2024-09-18T12:10:07.473 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-18T12:10:07.491 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.ojPvTQIdeJ 2024-09-18T12:10:07.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ojPvTQIdeJ 2024-09-18T12:10:07.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.ojPvTQIdeJ 2024-09-18T12:10:07.692 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.ojPvTQIdeJ is 291KB 2024-09-18T12:10:07.757 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.ojPvTQIdeJ 2024-09-18T12:10:07.763 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910993/data/mon.b.tgz 2024-09-18T12:10:07.763 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-18T12:10:07.770 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.m2nZ2ZBb6o 2024-09-18T12:10:07.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.m2nZ2ZBb6o 2024-09-18T12:10:07.894 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.m2nZ2ZBb6o 2024-09-18T12:10:07.918 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.m2nZ2ZBb6o is 296KB 2024-09-18T12:10:07.976 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.m2nZ2ZBb6o 2024-09-18T12:10:07.983 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910993/data/mon.c.tgz 2024-09-18T12:10:07.983 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-18T12:10:07.990 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.NheUdNA5eF 2024-09-18T12:10:07.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.NheUdNA5eF 2024-09-18T12:10:08.068 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.NheUdNA5eF 2024-09-18T12:10:08.093 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.NheUdNA5eF is 296KB 2024-09-18T12:10:08.153 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.NheUdNA5eF 2024-09-18T12:10:08.159 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-18T12:10:08.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T12:10:08.186 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-18T12:10:08.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T12:10:08.253 INFO:teuthology.orchestra.run.smithi154.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T12:10:08.260 INFO:tasks.ceph:Archiving crash dumps... 2024-09-18T12:10:08.262 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910993/remote/smithi154/crash 2024-09-18T12:10:08.263 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T12:10:08.278 INFO:tasks.ceph:Compressing logs... 2024-09-18T12:10:08.279 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T12:10:08.294 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46474.log 2024-09-18T12:10:08.294 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39822.log 2024-09-18T12:10:08.295 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44935.log 2024-09-18T12:10:08.295 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46474.log.gz 2024-09-18T12:10:08.296 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43287.log 2024-09-18T12:10:08.296 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39822.log.gz 2024-09-18T12:10:08.296 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46501.log 2024-09-18T12:10:08.298 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33418.log 2024-09-18T12:10:08.298 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44935.log.gz 2024-09-18T12:10:08.298 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43287.log.gz 2024-09-18T12:10:08.298 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47230.log 2024-09-18T12:10:08.298 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46501.log: /var/log/ceph/ceph-client.admin.33418.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.46501.log.gz/var/log/ceph/ceph-client.admin.48202.log 2024-09-18T12:10:08.298 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-18T12:10:08.298 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33418.log.gz 2024-09-18T12:10:08.299 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36860.log 2024-09-18T12:10:08.299 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47230.log.gz 2024-09-18T12:10:08.299 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48202.log.gz 2024-09-18T12:10:08.299 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30783.log 2024-09-18T12:10:08.300 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39498.log 2024-09-18T12:10:08.301 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36860.log.gz 2024-09-18T12:10:08.301 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43800.log 2024-09-18T12:10:08.301 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30783.log.gz 2024-09-18T12:10:08.301 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39498.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.43368.log -- replaced with /var/log/ceph/ceph-client.admin.39498.log.gz 2024-09-18T12:10:08.301 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-18T12:10:08.301 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43800.log.gz 2024-09-18T12:10:08.301 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39174.log 2024-09-18T12:10:08.301 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36374.log 2024-09-18T12:10:08.303 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43368.log: /var/log/ceph/ceph-client.admin.39174.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39174.log.gz 2024-09-18T12:10:08.303 INFO:teuthology.orchestra.run.smithi154.stderr:-5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.33094.log -- replaced with /var/log/ceph/ceph-client.admin.43368.log.gz 2024-09-18T12:10:08.303 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-18T12:10:08.303 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39228.log 2024-09-18T12:10:08.303 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36374.log.gz 2024-09-18T12:10:08.303 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32824.log 2024-09-18T12:10:08.303 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33094.log.gz 2024-09-18T12:10:08.303 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39390.log 2024-09-18T12:10:08.304 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39228.log.gz 2024-09-18T12:10:08.304 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35213.log 2024-09-18T12:10:08.305 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32824.log: /var/log/ceph/ceph-client.admin.39390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32824.log.gz 2024-09-18T12:10:08.305 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44502.log 2024-09-18T12:10:08.305 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39390.log.gz 2024-09-18T12:10:08.305 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36968.log 2024-09-18T12:10:08.306 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35213.log.gz 2024-09-18T12:10:08.306 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48418.log 2024-09-18T12:10:08.306 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44502.log.gz 2024-09-18T12:10:08.306 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36968.log.gz 2024-09-18T12:10:08.307 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38307.log 2024-09-18T12:10:08.307 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48418.log.gz 2024-09-18T12:10:08.307 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46231.log 2024-09-18T12:10:08.307 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21072.log 2024-09-18T12:10:08.307 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38307.log.gz 2024-09-18T12:10:08.307 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35159.log 2024-09-18T12:10:08.307 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46231.log.gz 2024-09-18T12:10:08.308 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.21072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40551.log 2024-09-18T12:10:08.309 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21072.log.gz 2024-09-18T12:10:08.309 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.35159.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33634.log 2024-09-18T12:10:08.309 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35159.log.gz 2024-09-18T12:10:08.309 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46825.log 2024-09-18T12:10:08.309 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40551.log.gz 2024-09-18T12:10:08.310 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33634.log.gz 2024-09-18T12:10:08.310 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23880.log 2024-09-18T12:10:08.310 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46825.log.gz 2024-09-18T12:10:08.310 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33661.log 2024-09-18T12:10:08.310 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.23880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36617.log 2024-09-18T12:10:08.310 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23880.log.gz 2024-09-18T12:10:08.312 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30607.log 2024-09-18T12:10:08.312 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33661.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33661.log.gz 2024-09-18T12:10:08.312 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25193.log 2024-09-18T12:10:08.313 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36617.log.gz 2024-09-18T12:10:08.313 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42365.log 2024-09-18T12:10:08.313 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30607.log.gz 2024-09-18T12:10:08.313 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34498.log 2024-09-18T12:10:08.313 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25193.log.gz 2024-09-18T12:10:08.313 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.42365.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27368.log 2024-09-18T12:10:08.313 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42365.log.gz 2024-09-18T12:10:08.314 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.34498.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.24258.log 2024-09-18T12:10:08.314 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34498.log.gz 2024-09-18T12:10:08.314 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41604.log 2024-09-18T12:10:08.314 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27368.log.gz 2024-09-18T12:10:08.314 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24258.log.gz 2024-09-18T12:10:08.314 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26149.log 2024-09-18T12:10:08.315 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41604.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.41604.log.gz--verbose 2024-09-18T12:10:08.315 INFO:teuthology.orchestra.run.smithi154.stderr:-- /var/log/ceph/ceph-client.admin.27315.log 2024-09-18T12:10:08.315 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44908.log 2024-09-18T12:10:08.315 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26149.log.gz 2024-09-18T12:10:08.316 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27315.log: /var/log/ceph/ceph-mon.a.log 2024-09-18T12:10:08.317 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27315.log.gz 2024-09-18T12:10:08.317 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.44908.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38712.log 2024-09-18T12:10:08.317 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44908.log.gz 2024-09-18T12:10:08.317 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24825.log 2024-09-18T12:10:08.317 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45664.log 2024-09-18T12:10:08.319 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38712.log.gz 2024-09-18T12:10:08.319 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30673.log 2024-09-18T12:10:08.319 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24825.log.gz 2024-09-18T12:10:08.319 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24312.log 2024-09-18T12:10:08.319 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45664.log.gz 2024-09-18T12:10:08.319 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30673.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30673.log.gz 2024-09-18T12:10:08.319 INFO:teuthology.orchestra.run.smithi154.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.32094.log 2024-09-18T12:10:08.319 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39417.log 2024-09-18T12:10:08.319 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24312.log.gz 2024-09-18T12:10:08.321 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38172.log 2024-09-18T12:10:08.321 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32094.log.gz 2024-09-18T12:10:08.321 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48445.log 2024-09-18T12:10:08.321 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39417.log.gz 2024-09-18T12:10:08.321 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24015.log 2024-09-18T12:10:08.322 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38172.log.gz 2024-09-18T12:10:08.322 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40065.log 2024-09-18T12:10:08.322 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48445.log.gz 2024-09-18T12:10:08.322 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24015.log.gz 2024-09-18T12:10:08.322 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38199.log 2024-09-18T12:10:08.322 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40065.log.gz 2024-09-18T12:10:08.322 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44854.log 2024-09-18T12:10:08.323 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30343.log 2024-09-18T12:10:08.324 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38199.log.gz 2024-09-18T12:10:08.324 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43206.log 2024-09-18T12:10:08.324 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44854.log.gz 2024-09-18T12:10:08.324 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31876.log 2024-09-18T12:10:08.324 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30343.log: /var/log/ceph/ceph-client.admin.43206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46366.log 2024-09-18T12:10:08.324 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30343.log.gz 2024-09-18T12:10:08.324 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43206.log.gz 2024-09-18T12:10:08.324 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.31876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23941.log 2024-09-18T12:10:08.324 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31876.log.gz 2024-09-18T12:10:08.327 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46042.log 2024-09-18T12:10:08.327 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46366.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46366.log.gz 2024-09-18T12:10:08.328 INFO:teuthology.orchestra.run.smithi154.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.41766.log 2024-09-18T12:10:08.328 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.23941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39606.log 2024-09-18T12:10:08.328 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23941.log.gz 2024-09-18T12:10:08.328 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46042.log.gz 2024-09-18T12:10:08.328 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-18T12:10:08.328 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41766.log.gz 2024-09-18T12:10:08.328 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29079.log 2024-09-18T12:10:08.328 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39606.log: /var/log/ceph/ceph-mon.b.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39606.log.gz 2024-09-18T12:10:08.328 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36428.log 2024-09-18T12:10:08.328 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38388.log 2024-09-18T12:10:08.328 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29079.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29079.log.gz 2024-09-18T12:10:08.329 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28559.log 2024-09-18T12:10:08.329 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36428.log.gz 2024-09-18T12:10:08.329 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30519.log /var/log/ceph/ceph-client.admin.38388.log: 2024-09-18T12:10:08.329 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38388.log.gz 2024-09-18T12:10:08.329 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40902.log 2024-09-18T12:10:08.329 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28559.log.gz 2024-09-18T12:10:08.329 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31768.log 2024-09-18T12:10:08.329 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30519.log.gz 2024-09-18T12:10:08.332 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45826.log 2024-09-18T12:10:08.332 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40902.log.gz 2024-09-18T12:10:08.332 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.31768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31768.log.gz 2024-09-18T12:10:08.332 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25147.log 2024-09-18T12:10:08.332 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45826.log.gz 2024-09-18T12:10:08.332 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29592.log 2024-09-18T12:10:08.332 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25147.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25147.log.gz 2024-09-18T12:10:08.332 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47176.log 2024-09-18T12:10:08.332 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42503.log 2024-09-18T12:10:08.332 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29592.log.gz 2024-09-18T12:10:08.332 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28614.log 2024-09-18T12:10:08.332 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47176.log.gz 2024-09-18T12:10:08.333 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42171.log 2024-09-18T12:10:08.333 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42503.log.gz 2024-09-18T12:10:08.333 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45259.log 2024-09-18T12:10:08.333 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28614.log.gz 2024-09-18T12:10:08.335 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47581.log 2024-09-18T12:10:08.335 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42171.log.gz 2024-09-18T12:10:08.335 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45259.log.gz 2024-09-18T12:10:08.335 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26369.log 2024-09-18T12:10:08.335 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47581.log.gz 2024-09-18T12:10:08.335 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26039.log 2024-09-18T12:10:08.335 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26369.log.gz 2024-09-18T12:10:08.336 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34768.log 2024-09-18T12:10:08.336 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32527.log 2024-09-18T12:10:08.336 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26039.log.gz 2024-09-18T12:10:08.336 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40173.log 2024-09-18T12:10:08.336 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34768.log.gz 2024-09-18T12:10:08.337 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.32527.log: -- /var/log/ceph/ceph-client.admin.34525.log 2024-09-18T12:10:08.337 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32527.log.gz 2024-09-18T12:10:08.337 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40173.log.gz 2024-09-18T12:10:08.337 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31714.log 2024-09-18T12:10:08.337 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 /var/log/ceph/ceph-client.admin.34525.log: --verbose -- /var/log/ceph/ceph-client.admin.33013.log 2024-09-18T12:10:08.338 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34525.log.gz 2024-09-18T12:10:08.338 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.31714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31714.log.gz 2024-09-18T12:10:08.338 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32932.log 2024-09-18T12:10:08.340 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33013.log.gz 2024-09-18T12:10:08.340 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25523.log 2024-09-18T12:10:08.340 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45367.log 2024-09-18T12:10:08.340 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32932.log: /var/log/ceph/ceph-client.admin.25523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32932.log.gz 2024-09-18T12:10:08.340 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25523.log.gz 2024-09-18T12:10:08.340 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40254.log 2024-09-18T12:10:08.340 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35969.log 2024-09-18T12:10:08.340 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45367.log: /var/log/ceph/ceph-client.admin.40254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40254.log.gz 2024-09-18T12:10:08.340 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39552.log 2024-09-18T12:10:08.341 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45367.log.gz 2024-09-18T12:10:08.341 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36779.log 2024-09-18T12:10:08.341 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39552.log: /var/log/ceph/ceph-client.admin.35834.log 2024-09-18T12:10:08.341 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35969.log.gz 2024-09-18T12:10:08.341 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39552.log.gz 2024-09-18T12:10:08.345 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33823.log 2024-09-18T12:10:08.345 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36779.log.gz 2024-09-18T12:10:08.345 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28934.log 2024-09-18T12:10:08.345 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37739.log 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35834.log.gz 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33823.log.gz 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29646.log 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28934.log.gz 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41307.log 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37739.log.gz 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28884.log 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29646.log: /var/log/ceph/ceph-client.admin.41307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29646.log.gz 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41307.log.gz 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48148.log 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28884.log.gz 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40524.log 2024-09-18T12:10:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38253.log 2024-09-18T12:10:08.347 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48148.log: /var/log/ceph/ceph-client.admin.40524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48148.log.gz 2024-09-18T12:10:08.347 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40524.log.gz 2024-09-18T12:10:08.347 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47878.log 2024-09-18T12:10:08.347 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38253.log.gz 2024-09-18T12:10:08.347 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39768.log 2024-09-18T12:10:08.347 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47878.log.gz 2024-09-18T12:10:08.347 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40875.log 2024-09-18T12:10:08.347 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39768.log.gz 2024-09-18T12:10:08.347 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44638.log 2024-09-18T12:10:08.347 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40875.log.gz 2024-09-18T12:10:08.353 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24854.log 2024-09-18T12:10:08.353 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44638.log.gz 2024-09-18T12:10:08.353 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41334.log 2024-09-18T12:10:08.353 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24854.log.gz 2024-09-18T12:10:08.353 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40632.log 2024-09-18T12:10:08.353 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29727.log 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41523.log 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41334.log.gz/var/log/ceph/ceph-client.admin.40632.log: 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39444.log 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40632.log.gz 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41523.log: gzip /var/log/ceph/ceph-client.admin.29727.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36644.log 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41523.log.gz 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29727.log.gz 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39444.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39444.log.gz 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36833.log 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36644.log.gz 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38658.log 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40227.log 2024-09-18T12:10:08.354 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36833.log: /var/log/ceph/ceph-client.admin.38658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38658.log.gz 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36833.log.gz 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39849.log 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40227.log.gz 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24096.log 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39849.log.gz 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25303.log 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25819.log 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24096.log.gz 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31930.log 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30717.log 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25303.log.gz 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46879.log 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25819.log: /var/log/ceph/ceph-client.admin.31930.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31930.log.gz -- replaced with /var/log/ceph/ceph-client.admin.25819.log.gz 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-18T12:10:08.355 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45475.log 2024-09-18T12:10:08.356 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30717.log.gz 2024-09-18T12:10:08.356 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46879.log.gz 2024-09-18T12:10:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41955.log 2024-09-18T12:10:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45475.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45475.log.gz-5 2024-09-18T12:10:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:--verbose -- /var/log/ceph/ceph-client.admin.47365.log 2024-09-18T12:10:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41847.log 2024-09-18T12:10:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41955.log.gz 2024-09-18T12:10:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38442.log 0.0% 2024-09-18T12:10:08.370 INFO:teuthology.orchestra.run.smithi154.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47365.log.gz 2024-09-18T12:10:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36590.log 2024-09-18T12:10:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41847.log: /var/log/ceph/ceph-client.admin.38442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25951.log 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38442.log.gz 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41847.log.gz 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33067.log 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36590.log.gz 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.25951.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48634.log 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25951.log.gz 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43152.log 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33067.log.gz 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26083.log 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48634.log: /var/log/ceph/ceph-client.admin.43152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24853.log 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48634.log.gz 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43152.log.gz 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.26083.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.20844.log 2024-09-18T12:10:08.371 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26083.log.gz 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24582.log 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24853.log.gz 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.20844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46312.log 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20844.log.gz 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.24582.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42692.log 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24582.log.gz 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41712.log 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46312.log.gz 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42063.log 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42692.log.gz 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24339.log 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41712.log.gz 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38577.log 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42063.log.gz 2024-09-18T12:10:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36455.log 2024-09-18T12:10:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24339.log.gz 2024-09-18T12:10:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44800.log 2024-09-18T12:10:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38577.log.gz 2024-09-18T12:10:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43044.log 2024-09-18T12:10:08.373 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36455.log.gz 2024-09-18T12:10:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25545.log 2024-09-18T12:10:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44800.log.gz 2024-09-18T12:10:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45799.log 2024-09-18T12:10:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43044.log.gz 2024-09-18T12:10:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34471.log 2024-09-18T12:10:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25545.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25545.log.gz 2024-09-18T12:10:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45799.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45799.log.gz-5 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:--verbose -- /var/log/ceph/ceph-client.admin.33850.log 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29565.log 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34471.log.gz 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25281.log 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33850.log.gz 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47716.log 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29565.log.gz 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24801.log 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25281.log.gz 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27419.log 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47716.log.gz 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30629.log 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24801.log.gz 2024-09-18T12:10:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36806.log 2024-09-18T12:10:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27419.log.gz 2024-09-18T12:10:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39633.log 2024-09-18T12:10:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30629.log.gz 2024-09-18T12:10:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36806.log.gz 2024-09-18T12:10:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35267.log 2024-09-18T12:10:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39633.log.gz 2024-09-18T12:10:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33742.log 2024-09-18T12:10:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29808.log 2024-09-18T12:10:08.375 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35267.log.gz 2024-09-18T12:10:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31687.log 2024-09-18T12:10:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33742.log.gz 2024-09-18T12:10:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33877.log 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29808.log: /var/log/ceph/ceph-client.admin.31687.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.29948.log -- replaced with /var/log/ceph/ceph-client.admin.29808.log.gz 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31687.log.gz 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29187.log 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33877.log.gz 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39984.log 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29948.log.gz 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42773.log 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29187.log.gz 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34633.log 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39984.log.gz 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37280.log 2024-09-18T12:10:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42773.log.gz 2024-09-18T12:10:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40767.log 2024-09-18T12:10:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34633.log.gz 2024-09-18T12:10:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24958.log 2024-09-18T12:10:08.377 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37280.log.gz 2024-09-18T12:10:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25325.log 2024-09-18T12:10:08.377 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40767.log.gz 2024-09-18T12:10:08.419 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41550.log 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24958.log.gz 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25435.log 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25325.log.gz 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34228.log 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41550.log.gz 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46933.log 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25435.log.gz 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37198.log 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34228.log.gz 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33931.log 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46933.log.gz 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39363.log 2024-09-18T12:10:08.420 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37198.log.gz 2024-09-18T12:10:08.421 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35240.log 2024-09-18T12:10:08.421 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33931.log.gz 2024-09-18T12:10:08.421 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38766.log 2024-09-18T12:10:08.421 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39363.log.gz 2024-09-18T12:10:08.421 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.35240.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28668.log 2024-09-18T12:10:08.421 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35240.log.gz 2024-09-18T12:10:08.421 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29511.log 2024-09-18T12:10:08.421 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38766.log.gz 2024-09-18T12:10:08.421 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36752.log 2024-09-18T12:10:08.421 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28668.log.gz 2024-09-18T12:10:08.421 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.17213.log 2024-09-18T12:10:08.421 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29511.log.gz 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32419.log 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36752.log.gz 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.tmp-client.admin.17213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46150.log 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.17213.log.gz 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24717.log 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32419.log.gz 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23942.log 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46150.log.gz 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.24717.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48499.log 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24717.log.gz 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37022.log 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.23942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23942.log.gz 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41901.log 2024-09-18T12:10:08.422 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48499.log.gz 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25995.log 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37022.log.gz 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48877.log 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41901.log.gz 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42530.log 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25995.log.gz 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47284.log 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48877.log.gz 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43638.log 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42530.log.gz 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26281.log 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47284.log.gz 2024-09-18T12:10:08.423 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37820.log 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43638.log.gz 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25699.log 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26281.log.gz 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26017.log 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37820.log.gz 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48310.log 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25699.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.47851.log -- replaced with /var/log/ceph/ceph-client.admin.25699.log.gz 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26017.log: /var/log/ceph/ceph-client.admin.48310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24186.log 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48310.log.gz 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26017.log.gz 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47770.log 2024-09-18T12:10:08.424 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47851.log.gz 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42144.log 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24186.log.gz 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28828.log 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47770.log.gz 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42144.log.gz 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33796.log 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30187.log 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28828.log.gz 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32283.log 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33796.log.gz 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.30187.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48769.log 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30187.log.gz 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41226.log 2024-09-18T12:10:08.425 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32283.log.gz 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24589.log 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48769.log.gz 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34660.log 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41226.log.gz 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48337.log 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24589.log.gz 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47662.log 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34660.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34660.log.gz 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42036.log 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48337.log.gz 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37712.log 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47662.log.gz 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43962.log 2024-09-18T12:10:08.426 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42036.log.gz 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39120.log 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37712.log.gz 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33769.log 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43962.log.gz 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40281.log 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39120.log.gz 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32229.log 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33769.log.gz 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44475.log 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40281.log.gz 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40794.log 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32229.log.gz 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37550.log 2024-09-18T12:10:08.427 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25171.log 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40794.log.gz 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35888.log 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37550.log.gz 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25863.log 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25171.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25171.log.gz 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35888.log.gz 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30497.log 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23807.log 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25863.log.gz 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32851.log 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30497.log.gz 2024-09-18T12:10:08.428 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26127.log 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.23807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23807.log.gz 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47095.log /var/log/ceph/ceph-client.admin.32851.log: 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32851.log.gz 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37685.log 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26127.log.gz 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46096.log 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47095.log.gz 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46177.log 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37685.log.gz 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40848.log 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46096.log.gz 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43530.log 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46177.log.gz 2024-09-18T12:10:08.429 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47149.log 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40848.log.gz 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24590.log 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43530.log.gz 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37658.log 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47149.log.gz 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23833.log 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24590.log.gz 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36401.log 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37658.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37658.log.gz-5 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr:--verbose -- /var/log/ceph/ceph-client.admin.38523.log 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.23833.log: /var/log/ceph/ceph-client.admin.36401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23833.log.gz 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 0.0%--verbose -- replaced with /var/log/ceph/ceph-client.admin.36401.log.gz 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr:-- /var/log/ceph/ceph-client.admin.30078.log 2024-09-18T12:10:08.430 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35753.log 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38523.log.gz 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25768.log 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30078.log.gz 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36995.log 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35753.log.gz 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37049.log 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25768.log.gz 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26259.log 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36995.log.gz 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44205.log 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37049.log.gz 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29892.log 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26259.log.gz 2024-09-18T12:10:08.431 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36320.log 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44205.log.gz 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29892.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29892.log.gzgzip 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.24181.log 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47419.log 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36320.log.gz 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32013.log 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24181.log.gz 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45745.log 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47419.log.gz 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47122.log 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32013.log.gz 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48175.log 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45745.log.gz 2024-09-18T12:10:08.432 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36212.log 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47122.log.gz 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48175.log: /var/log/ceph/ceph-client.admin.24426.log 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48175.log.gz 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44989.log 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36212.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36212.log.gz 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:-- /var/log/ceph/ceph-client.admin.34174.log 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24426.log.gz 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.44989.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32202.log 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44989.log.gz 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34147.log 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34174.log.gz 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48472.log 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32202.log.gz 2024-09-18T12:10:08.433 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35861.log 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34147.log.gz 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48472.log.gz 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33175.log 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35861.log.gz 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35024.log 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38334.log 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33175.log.gz 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37766.log 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35024.log.gz 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43098.log 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38334.log.gz 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37766.log.gz 2024-09-18T12:10:08.434 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45583.log 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40821.log 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43098.log.gz 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41172.log 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45583.log.gz 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43854.log 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40821.log.gz 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38901.log 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41172.log.gz 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43395.log 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43854.log.gz 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38064.log 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38901.log.gz 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26193.log 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43395.log.gz 2024-09-18T12:10:08.435 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41145.log 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38064.log.gz 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42665.log 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26193.log.gz 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41145.log.gz 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43233.log 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38685.log 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42665.log.gz 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44773.log 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43233.log.gz 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46447.log 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38685.log.gz 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41658.log 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44773.log.gz 2024-09-18T12:10:08.436 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42557.log 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46447.log.gz 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32067.log 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41658.log.gz 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25929.log 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42557.log.gz 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29214.log 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43476.log 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32067.log.gz 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25929.log.gz 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33337.log 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29214.log.gz 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40362.log 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43476.log.gz 2024-09-18T12:10:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44746.log 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33337.log.gz 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24663.log 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40362.log.gz 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45853.log 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44746.log: /var/log/ceph/ceph-client.admin.24663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44746.log.gz 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24663.log.gz 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30651.log 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34039.log 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45853.log.gz 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30651.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30651.log.gz 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34039.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34039.log.gz 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30761.log 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31849.log 2024-09-18T12:10:08.438 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40740.log 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30761.log.gz 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.31849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39741.log 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31849.log.gz 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48796.log 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40740.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40740.log.gz 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39741.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39741.log.gz 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48796.log.gz 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48013.log 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26391.log 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29943.log 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48013.log.gz 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42009.log 2024-09-18T12:10:08.439 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26391.log.gz 2024-09-18T12:10:08.440 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23853.log 2024-09-18T12:10:08.440 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29943.log.gz 2024-09-18T12:10:08.477 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42009.log.gz 2024-09-18T12:10:08.477 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33445.log 2024-09-18T12:10:08.477 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.23853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23853.log.gz 2024-09-18T12:10:08.477 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48229.log 2024-09-18T12:10:08.477 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33445.log.gz 2024-09-18T12:10:08.477 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27473.log 2024-09-18T12:10:08.477 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37117.log 2024-09-18T12:10:08.477 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44097.log 2024-09-18T12:10:08.477 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48229.log.gz 2024-09-18T12:10:08.477 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27473.log.gz 2024-09-18T12:10:08.477 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45124.log 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37117.log.gz 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28745.log 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44097.log.gz 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26105.log 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33229.log 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45124.log.gz 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28745.log.gz 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26105.log.gz 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30233.log 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35456.log 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33229.log: /var/log/ceph/ceph-client.admin.30233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33229.log.gz 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30233.log.gz 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25215.log 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31795.log 2024-09-18T12:10:08.478 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35456.log.gz 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25215.log.gz 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33391.log 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29754.log 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.31795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31795.log.gz 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44340.log 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33391.log.gz 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38415.log 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29754.log.gz 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35591.log 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44340.log.gz 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41280.log 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38415.log.gz 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35591.log.gz 2024-09-18T12:10:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38091.log 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41280.log.gz 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30299.log 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37983.log 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38091.log.gz 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40713.log 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30299.log.gz 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45070.log 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37983.log.gz 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40713.log.gz 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46258.log 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44286.log 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45070.log.gz 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35915.log 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46258.log.gz 2024-09-18T12:10:08.480 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33688.log 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44286.log.gz 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.35915.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37334.log 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35915.log.gz 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33121.log 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33688.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33688.log.gz 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46123.log 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37334.log.gz 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33121.log.gz 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37307.log 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32446.log 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46123.log.gz 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37144.log 2024-09-18T12:10:08.481 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24103.log 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37307.log.gz 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32446.log.gz 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37144.log.gz 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27284.log 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25907.log 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24103.log.gz 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45313.log 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27284.log: 0.0%/var/log/ceph/ceph-client.admin.25907.log: -- replaced with /var/log/ceph/ceph-client.admin.27284.log.gz 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38145.log 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25907.log.gz 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30431.log 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45313.log: /var/log/ceph/ceph-client.admin.38145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42117.log 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45313.log.gz 2024-09-18T12:10:08.482 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38145.log.gz 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32635.log 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30431.log.gz 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42117.log.gz 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46555.log 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32635.log.gz 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36131.log 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46555.log.gz 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38739.log 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47527.log /var/log/ceph/ceph-client.admin.36131.log: 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36131.log.gz 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42338.log 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38739.log: /var/log/ceph/ceph-client.admin.47527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38739.log.gz 2024-09-18T12:10:08.483 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33553.log 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47527.log.gz 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25633.log 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42338.log: /var/log/ceph/ceph-client.admin.33553.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.42338.log.gz--verbose 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr:-- /var/log/ceph/ceph-client.admin.39687.log 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33553.log.gz 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25633.log.gz 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32148.log 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39687.log.gz 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24962.log 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32148.log.gz 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42828.log 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24962.log.gz 2024-09-18T12:10:08.484 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40119.log 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27608.log 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42828.log.gz 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40119.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40119.log.gz 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.39660.log 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25589.log 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27608.log.gz 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40200.log 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39660.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39660.log.gz 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25589.log: /var/log/ceph/ceph-client.admin.42225.log 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25589.log.gz 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.40200.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27554.log 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40200.log.gz 2024-09-18T12:10:08.485 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29781.log 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42225.log.gz 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27554.log.gz 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46717.log 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40686.log 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29781.log.gz 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33958.log 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46717.log.gz 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37469.log 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40686.log: /var/log/ceph/ceph-client.admin.33958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29430.log 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40686.log.gz 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33958.log.gz 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr: 95.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-18T12:10:08.486 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25721.log 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37469.log.gz 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37902.log 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29430.log: /var/log/ceph/ceph-client.admin.25721.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29430.log.gz 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48985.log 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25721.log.gz 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42284.log 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37902.log.gz 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25841.log 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48985.log.gz 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40011.log 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42284.log.gz 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25567.log 2024-09-18T12:10:08.487 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25841.log.gz 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36293.log 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40011.log.gz 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44692.log 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25567.log.gz 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36293.log.gz 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27635.log 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45178.log 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44692.log: /var/log/ceph/ceph-client.admin.27635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24023.log 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44692.log.gz 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27635.log.gz 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43773.log 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45178.log.gz 2024-09-18T12:10:08.488 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35483.log 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24023.log.gz 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43773.log.gz 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45718.log 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35483.log.gz 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41820.log 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45718.log.gz 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33607.log 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27581.log 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41820.log.gz 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40956.log 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33607.log.gz 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27581.log.gz 2024-09-18T12:10:08.489 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33364.log 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36941.log 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40956.log.gz 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47905.log 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33364.log.gz 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48904.log 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36941.log: /var/log/ceph/ceph-client.admin.47905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36941.log.gz 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 0.0%--verbose -- replaced with /var/log/ceph/ceph-client.admin.47905.log.gz-- 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37253.log 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42473.log 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48904.log.gz 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25391.log 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37253.log.gz 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30277.log 2024-09-18T12:10:08.490 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42473.log.gz 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26061.log 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25391.log.gz 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35321.log 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30277.log: /var/log/ceph/ceph-client.admin.26061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30277.log.gz 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21046.log 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26061.log.gz 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29160.log 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35321.log.gz 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.21046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21046.log.gz 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40308.log 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29160.log.gz 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37225.log 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27500.log 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40308.log.gz 2024-09-18T12:10:08.491 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37225.log.gz 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48850.log 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45394.log 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27500.log.gz 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25973.log 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48850.log.gz 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47311.log 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45394.log.gz 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42936.log 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25973.log.gz 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47311.log.gz 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21133.log 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38982.log 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42936.log.gz 2024-09-18T12:10:08.492 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46690.log 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.21133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21133.log.gz 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41199.log 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38982.log.gz 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46690.log.gz 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45880.log 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43260.log 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41199.log.gz 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45880.log.gz 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28529.log 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43260.log.gz 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46015.log 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25885.log 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28529.log.gz 2024-09-18T12:10:08.493 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46420.log 2024-09-18T12:10:08.494 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46015.log.gz 2024-09-18T12:10:08.494 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25885.log.gz 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39201.log 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33526.log 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46420.log.gz 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26215.log 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39201.log.gz 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35078.log 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33526.log.gz 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39903.log 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25347.log 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26215.log.gz 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35078.log: gzip /var/log/ceph/ceph-client.admin.39903.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44827.log 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35078.log.gz 2024-09-18T12:10:08.518 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39903.log.gz 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29376.log 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25347.log.gz 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33715.log 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44827.log.gz 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29376.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29376.log.gz 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34957.log 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43719.log /var/log/ceph/ceph-client.admin.33715.log: 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33715.log.gz 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44232.log 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34957.log.gz 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40416.log 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43719.log.gz 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32338.log 2024-09-18T12:10:08.519 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44232.log.gz 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45961.log 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40416.log.gz 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36347.log 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32338.log.gz 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44421.log 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45961.log.gz 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47500.log 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36347.log.gz 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44421.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.35186.log -- replaced with /var/log/ceph/ceph-client.admin.44421.log.gz 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41928.log 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47500.log.gz 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35186.log.gzgzip 2024-09-18T12:10:08.520 INFO:teuthology.orchestra.run.smithi154.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.24104.log 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46906.log 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41928.log.gz 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25795.log 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24104.log.gz 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36914.log 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46906.log.gz 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36023.log 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25795.log.gz 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42990.log 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36914.log.gz 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28880.log 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36023.log.gz 2024-09-18T12:10:08.521 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42990.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42990.log.gz 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37388.log 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21097.log 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28880.log.gz 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37523.log 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37388.log.gz 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.21097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45340.log 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21097.log.gz 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43017.log 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37523.log.gz 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45340.log.gz 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30002.log 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43017.log.gz 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36077.log 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34255.log 2024-09-18T12:10:08.522 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30002.log: 0.0%/var/log/ceph/ceph-client.admin.36077.log: -- replaced with /var/log/ceph/ceph-client.admin.30002.log.gz 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36077.log.gz 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25016.log 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34255.log: /var/log/ceph/ceph-client.admin.25016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36725.log 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34255.log.gz 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25016.log.gz 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37793.log 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.36725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35564.log 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36725.log.gz 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45151.log 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37793.log.gz 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46663.log 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47689.log 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45151.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35564.log.gz 2024-09-18T12:10:08.523 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45151.log.gz 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44178.log 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46663.log.gz 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48607.log 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47689.log.gz 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41577.log 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44178.log.gz 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34579.log 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48607.log.gz 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41577.log.gz 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32040.log 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30695.log 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34579.log.gz 2024-09-18T12:10:08.524 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32040.log.gz 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42090.log 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34201.log 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30695.log: gzip -5 0.0%--verbose -- replaced with /var/log/ceph/ceph-client.admin.30695.log.gz-- 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.31822.log 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42090.log: /var/log/ceph/ceph-client.admin.34201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42090.log.gz 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24285.log 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34201.log.gz 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.31822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24509.log 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31822.log.gz 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25457.log 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24285.log.gz 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28938.log 2024-09-18T12:10:08.525 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24509.log.gz 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42638.log 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25457.log.gz 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35402.log 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28938.log.gz 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28664.log 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42638.log.gz 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37090.log 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35402.log.gz 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28664.log.gz 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29268.log 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37090.log.gz 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29484.log 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47554.log 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29268.log.gz 2024-09-18T12:10:08.526 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30051.log 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29484.log.gz 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47554.log.gz 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29403.log 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30051.log.gz 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41253.log 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34336.log 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29403.log.gz 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30541.log 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41253.log: /var/log/ceph/ceph-client.admin.34336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44043.log 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41253.log.gz 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34336.log.gz 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44556.log 2024-09-18T12:10:08.527 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30541.log.gz 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35807.log 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44043.log.gz 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24908.log 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44556.log.gz 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41739.log 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35807.log.gz 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46771.log 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42446.log 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24908.log.gz 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41739.log: /var/log/ceph/ceph-client.admin.46771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46285.log 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46771.log.gz 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41739.log.gz 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42446.log.gz 2024-09-18T12:10:08.528 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43179.log 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46285.log.gz 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42392.log 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.43179.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38820.log 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43179.log.gz 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42392.log.gz 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36482.log 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38820.log.gz 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33040.log 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36482.log.gz 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45232.log 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34714.log 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33040.log: /var/log/ceph/ceph-client.admin.45232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33040.log.gz 2024-09-18T12:10:08.529 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45232.log.gz 2024-09-18T12:10:08.546 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47824.log 2024-09-18T12:10:08.546 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48040.log 2024-09-18T12:10:08.546 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34714.log.gz 2024-09-18T12:10:08.546 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24177.log 2024-09-18T12:10:08.546 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47824.log.gz 2024-09-18T12:10:08.546 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41793.log 2024-09-18T12:10:08.546 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48040.log.gz 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41496.log 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24177.log: 0.0%/var/log/ceph/ceph-client.admin.41793.log: -- replaced with /var/log/ceph/ceph-client.admin.24177.log.gz 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30321.log 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41793.log.gz 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45421.log 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41496.log.gz 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30321.log.gz 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44259.log 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47338.log 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45421.log.gz 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43584.log 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44259.log.gz 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42746.log 2024-09-18T12:10:08.547 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47338.log.gz 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44665.log 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43584.log.gz 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42746.log.gz 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49010.log 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44665.log.gz 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25259.log 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.49010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49010.log.gz 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47014.log 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25259.log.gz 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34417.log 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48715.log 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47014.log.gz 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34930.log 2024-09-18T12:10:08.548 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34417.log.gz 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41982.log 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48715.log.gz 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34930.log.gz 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48823.log 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41631.log 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41982.log.gz 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36536.log 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48823.log.gz 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39093.log 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41631.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41631.log.gz 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32770.log 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36536.log.gz 2024-09-18T12:10:08.549 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28988.log 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39093.log: /var/log/ceph/ceph-client.admin.32770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39093.log.gz 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32770.log.gz 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43746.log 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28988.log: /var/log/ceph/ceph-client.admin.32121.log 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28988.log.gz 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34066.log 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43746.log.gz 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46339.log 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32121.log.gz 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34066.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.34066.log.gz-5 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:--verbose -- /var/log/ceph/ceph-client.admin.41442.log 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46339.log.gz 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43125.log 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27527.log 2024-09-18T12:10:08.550 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41442.log.gz 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28560.log 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43125.log.gz 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24366.log 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39795.log 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27527.log.gz 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28560.log.gz 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24021.log 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24366.log.gz 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39795.log.gz 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38631.log 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24021.log.gz 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36671.log 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48391.log 2024-09-18T12:10:08.551 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38631.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38631.log.gz 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35780.log 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36671.log.gz 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43071.log 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48391.log.gz 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44448.log 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35780.log: /var/log/ceph/ceph-client.admin.43071.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.40146.log 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35780.log.gz 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43071.log.gz 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28502.log 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44448.log.gz 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30453.log 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40146.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40146.log.gz 2024-09-18T12:10:08.552 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46069.log 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28502.log.gz 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30453.log.gz 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23936.log 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40092.log 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46069.log: /var/log/ceph/ceph-client.admin.23936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46069.log.gz 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23936.log.gz 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45043.log 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41415.log 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40092.log.gz 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45043.log.gz 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47932.log 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25066.log 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41415.log.gz 2024-09-18T12:10:08.553 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42909.log 2024-09-18T12:10:08.554 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25066.log: /var/log/ceph/ceph-client.admin.47932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25066.log.gz 2024-09-18T12:10:08.554 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37631.log 2024-09-18T12:10:08.554 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47932.log.gz 2024-09-18T12:10:08.554 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45529.log 2024-09-18T12:10:08.572 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42909.log: /var/log/ceph/ceph-client.admin.37631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25120.log 2024-09-18T12:10:08.572 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37631.log.gz 2024-09-18T12:10:08.572 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42909.log.gz 2024-09-18T12:10:08.572 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34093.log 2024-09-18T12:10:08.572 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45529.log.gz 2024-09-18T12:10:08.572 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25369.log 2024-09-18T12:10:08.572 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25120.log.gz 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34093.log.gz 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47635.log 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25369.log.gz 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30255.log 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32175.log 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47635.log.gz 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30255.log.gz 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37956.log 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32175.log.gz 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30739.log 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48553.log 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37956.log.gz 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33202.log 2024-09-18T12:10:08.573 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30739.log.gz 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38226.log 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48553.log.gz 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30159.log 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33202.log.gz 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48742.log 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38226.log.gz 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30159.log.gz 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39930.log 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44584.log 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48742.log: /var/log/ceph/ceph-client.admin.39930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48742.log.gz 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39930.log.gz 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27423.log 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44584.log.gz 2024-09-18T12:10:08.574 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28610.log 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27423.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27423.log.gz 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44881.log 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28610.log.gz 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32689.log 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44016.log 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44881.log.gz 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46744.log 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32689.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.32689.log.gz--verbose 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr:-- /var/log/ceph/ceph-client.admin.41010.log 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44016.log.gz 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34282.log 0.0% 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46744.log.gz 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39147.log 2024-09-18T12:10:08.575 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41037.log 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41010.log.gz 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34282.log.gz 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39147.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39147.log.gz 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42311.log 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44719.log 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41037.log.gz 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35996.log 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42311.log.gz 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32581.log 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44719.log.gz 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42963.log 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35996.log.gz 2024-09-18T12:10:08.576 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34849.log 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32581.log.gz 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29997.log 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42963.log.gz 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35294.log 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34849.log.gz 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43665.log 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29997.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29997.log.gz 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46798.log 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35294.log.gz 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32905.log 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43665.log.gz 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46798.log.gz 2024-09-18T12:10:08.577 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38793.log 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29106.log 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32905.log.gz 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26171.log 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38793.log.gz 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29106.log.gz 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39579.log 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26171.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26171.log.gz 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32716.log 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39579.log.gz 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44529.log 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34822.log 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32716.log.gz 2024-09-18T12:10:08.578 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39714.log 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44529.log.gz 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48256.log 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46852.log 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34822.log.gz 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39714.log: /var/log/ceph/ceph-client.admin.48256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39714.log.gz 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37442.log 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48256.log.gz 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38604.log 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46852.log.gz 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23907.log 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37442.log.gz 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35348.log 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38604.log.gz 2024-09-18T12:10:08.579 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20989.log 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.23907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23907.log.gz 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47608.log 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35348.log.gz 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44070.log 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.20989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20989.log.gz 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46528.log 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47608.log.gz 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45286.log 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44070.log.gz 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43449.log 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46528.log.gz 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30585.log 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45286.log.gz 2024-09-18T12:10:08.580 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35429.log 2024-09-18T12:10:08.581 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43449.log.gz 2024-09-18T12:10:08.581 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30585.log.gz 2024-09-18T12:10:08.601 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47446.log 2024-09-18T12:10:08.601 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28799.log 2024-09-18T12:10:08.601 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35429.log.gz 2024-09-18T12:10:08.601 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42611.log 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47446.log.gz 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28799.log.gz 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48661.log 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40038.log 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42611.log.gz 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34741.log 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48661.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48661.log.gz 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41064.log 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40038.log.gz 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26325.log 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34741.log.gz 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35132.log 2024-09-18T12:10:08.602 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41064.log.gz 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42882.log 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26325.log.gz 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30409.log 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35132.log.gz 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34687.log 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42882.log.gz 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37415.log 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32311.log 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30409.log.gz 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45610.log 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34687.log.gz 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37415.log.gz 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30105.log 2024-09-18T12:10:08.603 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32311.log.gz 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45637.log 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36887.log 0.0% 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45610.log.gz 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30105.log.gz 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35645.log 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45637.log.gz 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31741.log 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36887.log.gz 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35645.log: /var/log/ceph/ceph-client.admin.44151.log 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35645.log.gz 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.31741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32959.log 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31741.log.gz 2024-09-18T12:10:08.604 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29013.log 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42719.log 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44151.log.gz 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32959.log.gz 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29013.log.gz 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39282.log 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32662.log 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42719.log.gz 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47068.log 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39282.log.gz 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.32662.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38361.log 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32662.log.gz 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34444.log 2024-09-18T12:10:08.605 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47068.log.gz 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31903.log 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38361.log.gz 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43692.log 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34444.log.gz 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32608.log 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.31903.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.31903.log.gz--verbose -- 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40605.log 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29133.log 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43692.log.gz 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32608.log.gz 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40605.log.gz 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37848.log 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40659.log 2024-09-18T12:10:08.606 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29133.log.gz 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37929.log 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37848.log.gz 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46960.log 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40659.log.gz 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45772.log 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37929.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.37929.log.gz--verbose 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr:-- /var/log/ceph/ceph-client.admin.41118.log 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40389.log 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46960.log.gz 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45772.log.gz 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36239.log 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41118.log.gz 2024-09-18T12:10:08.607 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 /var/log/ceph/ceph-client.admin.40389.log: --verbose -- /var/log/ceph/ceph-client.admin.32256.log 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40389.log.gz 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24428.log 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36239.log.gz 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48688.log 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32256.log.gz 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46609.log 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45097.log 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24428.log.gz 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48688.log: /var/log/ceph/ceph-client.admin.46609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48688.log.gz 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35537.log 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46609.log.gz 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24505.log 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45097.log.gz 2024-09-18T12:10:08.608 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28830.log 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35537.log.gz 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43341.log 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24505.log.gz 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25012.log 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28830.log.gz 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34012.log 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43341.log.gz 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.25012.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36185.log 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25012.log.gz 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32797.log 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34012.log.gz 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36185.log.gz 2024-09-18T12:10:08.609 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35672.log 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32797.log.gz 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30563.log 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40470.log 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35672.log.gz 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30563.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30563.log.gz 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39009.log 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40470.log.gz 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47743.log 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40983.log 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39009.log.gz 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32878.log 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47743.log.gz 2024-09-18T12:10:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31986.log 2024-09-18T12:10:08.611 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40983.log.gz 2024-09-18T12:10:08.611 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35375.log 2024-09-18T12:10:08.611 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32878.log.gz 2024-09-18T12:10:08.611 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44313.log 2024-09-18T12:10:08.611 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.31986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31986.log.gz 2024-09-18T12:10:08.611 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46636.log 2024-09-18T12:10:08.611 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35375.log.gz 2024-09-18T12:10:08.611 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24774.log 2024-09-18T12:10:08.611 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44313.log.gz 2024-09-18T12:10:08.611 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46636.log.gz 2024-09-18T12:10:08.631 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35105.log 2024-09-18T12:10:08.631 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43611.log 2024-09-18T12:10:08.631 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24774.log.gz 2024-09-18T12:10:08.631 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24504.log 2024-09-18T12:10:08.631 INFO:teuthology.orchestra.run.smithi154.stderr: 95.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35105.log.gz 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24427.log 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43611.log.gz 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34363.log 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24504.log.gz 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24427.log.gz 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32986.log 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27314.log 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34363.log.gz 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39066.log 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32986.log.gz 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-18T12:10:08.632 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27314.log.gz 2024-09-18T12:10:08.633 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38955.log 2024-09-18T12:10:08.633 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39066.log.gz 2024-09-18T12:10:08.633 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48526.log 2024-09-18T12:10:08.633 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39525.log 2024-09-18T12:10:08.633 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38955.log.gz 2024-09-18T12:10:08.633 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36563.log 2024-09-18T12:10:08.633 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48526.log.gz 2024-09-18T12:10:08.633 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42252.log 2024-09-18T12:10:08.633 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39525.log.gz 2024-09-18T12:10:08.633 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41091.log 2024-09-18T12:10:08.633 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36563.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36563.log.gz 2024-09-18T12:10:08.633 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34876.log 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42252.log.gz 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25677.log 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41091.log.gz 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34876.log.gz 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34606.log 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25677.log.gz 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44611.log 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34606.log.gz 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45556.log 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29893.log 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33499.log 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42800.log 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45556.log.gz 2024-09-18T12:10:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47473.log 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44611.log.gz 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29893.log.gz 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33499.log.gz 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48580.log 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42800.log.gz 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39957.log 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47473.log.gz 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48580.log.gz 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32743.log 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39957.log.gz 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48067.log 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32743.log.gz 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32500.log 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41469.log 2024-09-18T12:10:08.635 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48067.log.gz 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35051.log 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32500.log: /var/log/ceph/ceph-client.admin.41469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32500.log.gz 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41469.log.gz 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47203.log 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33904.log 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44124.log 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35051.log.gz 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47203.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47203.log.gz 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33985.log 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33904.log.gz 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38550.log 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44124.log.gz 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48094.log 2024-09-18T12:10:08.636 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33985.log: /var/log/ceph/ceph-client.admin.38550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33985.log.gz 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47392.log 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38550.log.gz 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39255.log 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30365.log 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48094.log.gz 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47392.log.gz 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24393.log /var/log/ceph/ceph-client.admin.39255.log: 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39255.log.gz 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35510.log 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30365.log.gz 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24393.log: gzip -5 0.0%--verbose -- /var/log/ceph/ceph-client.admin.43314.log 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr: -- replaced with /var/log/ceph/ceph-client.admin.24393.log.gz 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42855.log 2024-09-18T12:10:08.637 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35510.log.gz 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37875.log 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43314.log: /var/log/ceph/ceph-client.admin.42855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42855.log.gz 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43314.log.gz 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44962.log 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43557.log 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26237.log 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37875.log.gz 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44962.log: 0.0%/var/log/ceph/ceph-client.admin.43557.log: -- replaced with /var/log/ceph/ceph-client.admin.44962.log.gz 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43557.log.gz 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45934.log 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29700.log 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26237.log.gz 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27257.log 2024-09-18T12:10:08.638 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45934.log.gz 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29700.log.gz 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29349.log 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33472.log 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27257.log.gz 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28718.log 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29349.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.25479.log 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29349.log.gz 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33472.log.gz 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44394.log 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28718.log.gz 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.25479.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45205.log 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25479.log.gz 2024-09-18T12:10:08.639 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40578.log 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44394.log.gz 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29322.log 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39876.log 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45205.log.gz 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40578.log.gz 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29862.log 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29322.log.gz 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30211.log 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39876.log.gz 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43935.log 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29862.log.gz 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24904.log 2024-09-18T12:10:08.640 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30211.log.gz 2024-09-18T12:10:08.641 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36266.log 2024-09-18T12:10:08.641 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43935.log.gz 2024-09-18T12:10:08.641 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47986.log 2024-09-18T12:10:08.641 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24904.log.gz 2024-09-18T12:10:08.641 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33256.log 2024-09-18T12:10:08.641 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36266.log.gz 2024-09-18T12:10:08.641 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47986.log.gz 2024-09-18T12:10:08.641 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42419.log 2024-09-18T12:10:08.641 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41874.log 2024-09-18T12:10:08.650 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37577.log 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42419.log.gz 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33256.log.gz 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41874.log.gz 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45907.log 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48283.log 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37577.log.gz 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45448.log 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45907.log.gz 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37604.log 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48283.log.gz 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45691.log 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46393.log 2024-09-18T12:10:08.651 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45448.log.gz 2024-09-18T12:10:08.652 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45691.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.37604.log.gz--verbose 2024-09-18T12:10:08.652 INFO:teuthology.orchestra.run.smithi154.stderr:-- /var/log/ceph/ceph-client.admin.38010.log 2024-09-18T12:10:08.652 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45691.log.gz 2024-09-18T12:10:08.652 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36698.log 2024-09-18T12:10:08.652 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46393.log.gz 2024-09-18T12:10:08.652 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38010.log.gz 2024-09-18T12:10:08.652 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35726.log 2024-09-18T12:10:08.652 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24747.log 2024-09-18T12:10:08.652 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36698.log.gz 2024-09-18T12:10:08.652 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35726.log.gz 2024-09-18T12:10:08.652 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38847.log 2024-09-18T12:10:08.652 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25093.log 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24747.log.gz 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29457.log 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38847.log.gz 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44367.log 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25093.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.25093.log.gz-- 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43827.log 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29457.log.gz 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.44367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44367.log.gz 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32473.log 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34795.log 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38037.log 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43827.log.gz 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32473.log.gz 2024-09-18T12:10:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34795.log.gz 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23781.log 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47797.log 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38037.log.gz 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46582.log 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.23781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23781.log.gz 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34309.log 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47797.log.gz 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39471.log 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46582.log.gz 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45988.log 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34309.log.gz 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21212.log 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39471.log.gz 2024-09-18T12:10:08.654 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34997.log 2024-09-18T12:10:08.655 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45988.log.gz 2024-09-18T12:10:08.655 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38118.log 2024-09-18T12:10:08.655 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.21212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21212.log.gz 2024-09-18T12:10:08.655 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34997.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34997.log.gz 2024-09-18T12:10:08.655 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43422.log 2024-09-18T12:10:08.655 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38118.log.gz 2024-09-18T12:10:08.658 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30808.log 2024-09-18T12:10:08.658 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33283.log 2024-09-18T12:10:08.658 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43422.log.gz 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37496.log 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30808.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30808.log.gz 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33283.log.gz 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30132.log 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37496.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.25611.log -- replaced with /var/log/ceph/ceph-client.admin.37496.log.gz 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40335.log 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42584.log 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30132.log.gz 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25611.log.gz 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27369.log 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40335.log.gz 2024-09-18T12:10:08.659 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42584.log.gz 2024-09-18T12:10:08.660 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43989.log 2024-09-18T12:10:08.660 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.27369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27369.log.gz 2024-09-18T12:10:08.660 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36509.log 2024-09-18T12:10:08.664 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38280.log 2024-09-18T12:10:08.664 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43989.log.gz 2024-09-18T12:10:08.664 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40497.log 2024-09-18T12:10:08.664 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36509.log.gz/var/log/ceph/ceph-client.admin.38280.log: 2024-09-18T12:10:08.664 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29538.log 2024-09-18T12:10:08.664 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38280.log.gz 2024-09-18T12:10:08.664 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35699.log 2024-09-18T12:10:08.664 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40497.log.gz 2024-09-18T12:10:08.664 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24690.log 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29538.log.gz 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35699.log.gz 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34390.log 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33310.log 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.24690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24690.log.gz 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46204.log 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34390.log.gz 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33310.log.gz 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28772.log 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43908.log 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46204.log.gz 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26303.log 2024-09-18T12:10:08.665 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28772.log.gz 2024-09-18T12:10:08.666 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.43908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43908.log.gz 2024-09-18T12:10:08.670 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29619.log 2024-09-18T12:10:08.671 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26303.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.26303.log.gz-5 2024-09-18T12:10:08.671 INFO:teuthology.orchestra.run.smithi154.stderr:--verbose -- /var/log/ceph/ceph-client.admin.40443.log 2024-09-18T12:10:08.671 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48364.log 2024-09-18T12:10:08.672 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37171.log 2024-09-18T12:10:08.672 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29619.log.gz 2024-09-18T12:10:08.672 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40443.log.gz 2024-09-18T12:10:08.673 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48364.log.gz 2024-09-18T12:10:08.675 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32392.log 2024-09-18T12:10:08.675 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38496.log 2024-09-18T12:10:08.675 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37171.log.gz 2024-09-18T12:10:08.677 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25237.log 2024-09-18T12:10:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33580.log 2024-09-18T12:10:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32392.log.gz 2024-09-18T12:10:08.678 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38496.log.gz 2024-09-18T12:10:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36104.log 2024-09-18T12:10:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25237.log.gz 2024-09-18T12:10:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25743.log 2024-09-18T12:10:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33580.log.gz 2024-09-18T12:10:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39036.log 2024-09-18T12:10:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36104.log.gz 2024-09-18T12:10:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39336.log 2024-09-18T12:10:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25743.log.gz 2024-09-18T12:10:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46987.log 2024-09-18T12:10:08.679 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39036.log: 0.0%/var/log/ceph/ceph-client.admin.39336.log: -- replaced with /var/log/ceph/ceph-client.admin.39036.log.gz 2024-09-18T12:10:08.679 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39336.log.gz 2024-09-18T12:10:08.680 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36158.log 2024-09-18T12:10:08.680 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.46987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47959.log 2024-09-18T12:10:08.680 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46987.log.gz 2024-09-18T12:10:08.680 INFO:teuthology.orchestra.run.smithi154.stderr:gzip /var/log/ceph/ceph-client.admin.36158.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45016.log 2024-09-18T12:10:08.680 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36158.log.gz 2024-09-18T12:10:08.681 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47959.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.47959.log.gz-5 2024-09-18T12:10:08.681 INFO:teuthology.orchestra.run.smithi154.stderr:--verbose -- /var/log/ceph/ceph-client.admin.37361.log 2024-09-18T12:10:08.681 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.45016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45016.log.gz 2024-09-18T12:10:08.682 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29052.log 2024-09-18T12:10:08.683 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43881.log 2024-09-18T12:10:08.683 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37361.log.gz 2024-09-18T12:10:08.683 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29835.log 2024-09-18T12:10:08.685 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29052.log: /var/log/ceph/ceph-client.admin.43881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43881.log.gz 2024-09-18T12:10:08.685 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48958.log 2024-09-18T12:10:08.685 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29052.log.gz 2024-09-18T12:10:08.686 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34903.log 2024-09-18T12:10:08.686 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29835.log.gz 2024-09-18T12:10:08.686 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45502.log 2024-09-18T12:10:08.686 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48958.log.gz 2024-09-18T12:10:08.686 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47257.log 2024-09-18T12:10:08.686 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33148.log 2024-09-18T12:10:08.686 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34903.log: /var/log/ceph/ceph-client.admin.45502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34903.log.gz 2024-09-18T12:10:08.686 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45502.log.gz 2024-09-18T12:10:08.686 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47257.log.gz 2024-09-18T12:10:08.688 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41388.log 2024-09-18T12:10:08.688 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.33148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25655.log 2024-09-18T12:10:08.688 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33148.log.gz 2024-09-18T12:10:08.688 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43503.log 2024-09-18T12:10:08.688 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39309.log 2024-09-18T12:10:08.688 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41388.log: /var/log/ceph/ceph-client.admin.25655.log: /var/log/ceph/ceph-client.admin.43503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41388.log.gz 2024-09-18T12:10:08.688 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25655.log.gz 2024-09-18T12:10:08.688 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43503.log.gz 2024-09-18T12:10:08.689 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48931.log 2024-09-18T12:10:08.694 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34120.log 2024-09-18T12:10:08.694 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.39309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39309.log.gz 2024-09-18T12:10:08.694 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31959.log 2024-09-18T12:10:08.694 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38874.log 2024-09-18T12:10:08.694 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48931.log.gz 2024-09-18T12:10:08.694 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25501.log 2024-09-18T12:10:08.694 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34120.log.gz 2024-09-18T12:10:08.694 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.31959.log: /var/log/ceph/ceph-client.admin.38874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31959.log.gz 2024-09-18T12:10:08.694 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38469.log 2024-09-18T12:10:08.694 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38874.log.gz 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36050.log 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25501.log.gz 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38469.log.gz 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34552.log 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29241.log 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36050.log.gz 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29673.log 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.34552.log: /var/log/ceph/ceph-client.admin.29241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34552.log.gz 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42198.log 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29241.log.gz 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30475.log 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47041.log 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.42198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29673.log.gz 2024-09-18T12:10:08.695 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42198.log.gz 2024-09-18T12:10:08.696 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30387.log /var/log/ceph/ceph-client.admin.30475.log: 2024-09-18T12:10:08.696 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30475.log.gz 2024-09-18T12:10:08.696 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.47041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47041.log.gz 2024-09-18T12:10:08.699 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25413.log 2024-09-18T12:10:08.699 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41361.log 2024-09-18T12:10:08.699 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.30387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30387.log.gz 2024-09-18T12:10:08.699 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.25413.log: -- /var/log/ceph/ceph-client.admin.41685.log 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25413.log.gz 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35942.log 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41361.log.gz 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.41685.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41685.log.gz 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26347.log 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35942.log.gz 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32365.log 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48121.log 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.26347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26347.log.gz 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29295.log 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32365.log.gz 2024-09-18T12:10:08.700 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32554.log 2024-09-18T12:10:08.701 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.48121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48121.log.gz 2024-09-18T12:10:08.701 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35618.log 2024-09-18T12:10:08.701 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29295.log.gz 2024-09-18T12:10:08.701 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.32554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32554.log.gz 2024-09-18T12:10:08.701 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38928.log 2024-09-18T12:10:08.702 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.35618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35618.log.gz 2024-09-18T12:10:08.702 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40929.log 2024-09-18T12:10:08.703 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38928.log.gz 2024-09-18T12:10:08.703 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.40929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40929.log.gz 2024-09-18T12:10:08.737 INFO:teuthology.orchestra.run.smithi154.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-18T12:10:08.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-18T12:10:09.044 INFO:teuthology.orchestra.run.smithi154.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-18T12:10:09.416 INFO:teuthology.orchestra.run.smithi154.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-18T12:10:10.030 INFO:teuthology.orchestra.run.smithi154.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-18T12:12:04.803 INFO:teuthology.orchestra.run.smithi154.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-18T12:13:13.344 INFO:teuthology.orchestra.run.smithi154.stderr: 90.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-18T12:14:11.119 INFO:teuthology.orchestra.run.smithi154.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-18T12:14:11.121 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-18T12:14:11.122 INFO:teuthology.orchestra.run.smithi154.stderr:real 4m2.835s 2024-09-18T12:14:11.122 INFO:teuthology.orchestra.run.smithi154.stderr:user 5m9.139s 2024-09-18T12:14:11.122 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m11.972s 2024-09-18T12:14:11.123 INFO:tasks.ceph:Archiving logs... 2024-09-18T12:14:11.124 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910993/remote/smithi154/log 2024-09-18T12:14:11.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T12:15:07.568 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T12:15:07.580 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T12:15:07.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T12:15:07.649 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T12:15:07.660 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T12:15:07.660 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T12:15:07.764 INFO:teuthology.orchestra.run.smithi154.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T12:15:07.764 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================== 2024-09-18T12:15:07.764 INFO:teuthology.orchestra.run.smithi154.stdout:*hv01.front.sepi 67.205.162.81 3 u 2 64 377 0.054 -1.473 0.256 2024-09-18T12:15:07.764 INFO:teuthology.orchestra.run.smithi154.stdout:+hv02.front.sepi 63.231.80.2 3 u 63 64 377 0.087 0.128 0.304 2024-09-18T12:15:07.764 INFO:teuthology.orchestra.run.smithi154.stdout:+hv03.front.sepi 74.6.168.72 3 u 5 64 377 0.085 -2.776 0.237 2024-09-18T12:15:07.764 INFO:teuthology.orchestra.run.smithi154.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-18T12:15:07.765 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T12:15:07.774 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T12:15:07.775 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T12:15:07.783 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T12:15:07.792 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T12:15:07.801 INFO:teuthology.task.internal:Duration was 1790.946659 seconds 2024-09-18T12:15:07.801 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T12:15:07.810 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T12:15:07.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T12:15:07.943 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T12:15:07.943 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-09-18T12:15:07.943 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T12:15:07.958 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T12:15:07.958 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T12:15:08.000 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T12:15:08.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T12:15:08.241 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T12:15:08.252 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T12:15:08.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T12:15:08.281 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T12:15:08.291 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:15:08.326 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-09-18T12:15:08.377 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T12:15:08.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T12:15:08.423 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T12:15:08.433 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T12:15:08.434 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910993/remote/smithi154 2024-09-18T12:15:08.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T12:15:08.458 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T12:15:08.459 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T12:15:08.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T12:15:08.474 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T12:15:08.474 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T12:15:08.484 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T12:15:08.484 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T12:15:08.513 INFO:teuthology.orchestra.run.smithi154.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 12:15 /home/ubuntu/cephtest 2024-09-18T12:15:08.515 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T12:15:08.527 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T12:15:08.547 INFO:teuthology.run:Summary data: description: rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} duration: 1790.9466588497162 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-18T12:15:08.548 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T12:15:08.621 INFO:teuthology.run:pass